Prompt Injection via Span-Identity Confusion (and Semantic Overlays Defense)
First seen Aug 31, 2026 · Updated Aug 31, 2026
This is a defensive research paper, not an active exploit report. It describes 'Semantic Overlays,' a technique using trained adapters on a model's residual stream to help LLMs distinguish trusted instructions from untrusted content (e.g., tool output, user input), thereby mitigating prompt injection attacks. The underlying threat it addresses—LLMs losing track of span identity and being manipulated by text that mimics instructions—is a well-known, genuine issue, but the raw data itself presents a mitigation, not a novel attack.
Technical Analysis
The paper's core observation is that all LLM input is undifferentiated tokens, so despite the serving stack knowing which spans are user input, tool output, or instructions, this metadata is not intrinsically communicated to the model, making it possible for adversarial text to impersonate instructions within any span. Prompt injection exploits this by embedding imperative-style text in untrusted context (e.g., tool outputs, document contents, web page text) that the model then executes as if it were a legitimate instruction. Semantic Overlays add an out-of-band signal at chosen prefill positions via learned adapters to mark spans as, for example, 'non-executable,' improving the model's ability to reject embedded instructions even when the text is designed to look authoritative. This crosses agent/tool boundaries wherever an agent ingests external content (tool results, retrieved documents, other agents' messages) and treats it as a channel through which instructions could be smuggled to hijack the model's behavior.
Detection Signatures
- Embedded imperative language within tool outputs, retrieved documents, or third-party content (e.g., phrases instructing the model to ignore prior instructions, change behavior, or execute new commands)
- Text in non-instruction spans (tool results, user-supplied documents, inter-agent messages) that mimics system/developer instruction formatting
- Sudden shifts in model output behavior correlating with ingestion of external/untrusted content
- Benchmarks referenced for evaluating susceptibility: SEP, TensorTrust, AlpacaFarm, PIArena
Remediation Steps
- 1
Adopt span-identity annotation techniques
Where feasible, use out-of-band or non-textual signaling (such as the Semantic Overlays approach or equivalent architectural mitigations) to help models distinguish trusted instructions from untrusted content spans.
- 2
Sanitize and isolate untrusted content
Treat all tool outputs, retrieved documents, and inter-agent messages as untrusted data; avoid concatenating them directly into instruction-bearing prompt regions without clear delimiters or filtering.
- 3
Apply prompt injection benchmarking
Regularly test agent pipelines against established prompt injection benchmarks (SEP, TensorTrust, AlpacaFarm, PIArena) to measure susceptibility and validate mitigations.
- 4
Layer defenses
Combine model-level mitigations (like overlays/steering) with system-level controls such as least-privilege tool access, output validation, and human review for high-impact actions.
Industries Most Exposed
Respond to this threat
Pro subscribers get a full AI-generated incident-response playbook for this threat — detection, containment, eradication, and recovery steps — plus an unlimited AI Threat Advisor for questions about your environment.