Attnlocate: Attention-Based Runtime Localization of Behavior-Guiding Instructions in LLM Agents

Technical Analysis

The paper addresses the well-known risk that LLM agents parse untrusted external data (tool outputs, retrieved documents, etc.) through the same natural-language channel used for legitimate instructions, allowing injected content to be dynamically interpreted as behavior-guiding directives during inference. Attnlocate treats localization of these influential spans as an object-detection problem: it aggregates multi-head, multi-layer attention signals into a token-level feature space and applies a 1-D U-Net with an anchor-free detection head to find attention 'activation traces' correlated with tool-invocation decisions. Detected spans are then adjudicated based on the authority/trust level of their originating source (e.g., untrusted tool output vs. system prompt), enabling dynamic blocking of malicious invocations without requiring static input/output filtering or retraining. This is evaluated across ten agent configurations and five LLM families against indirect prompt injection and tool-poisoning scenarios, reporting strong detection metrics (mean IoU 0.743, AUROC 0.956, TPR 0.934 at FPR 0.067) and cross-model transferability.

Detection Signatures

• N/A - this is a defensive detection framework, not an attack technique.
• Relevant indicators the framework itself is designed to catch: unexpected tool-call arguments or invocations correlating with attention spikes on low-authority/untrusted context spans; tool outputs or retrieved documents containing embedded imperative instructions (e.g., 'ignore previous instructions', 'call function X with parameter Y') that shift downstream model attention disproportionately relative to their source authority level.

Remediation Steps

1. 1

   Evaluate runtime attention-based detectors

   Consider integrating attention-localization approaches like Attnlocate as a complementary runtime defense layer to detect dynamically injected behavior-guiding instructions in agent pipelines.

2. 2

   Enforce authority/provenance tagging

   Tag all context spans (system prompt, user input, tool output, retrieved documents) with a trust/authority level and use this to gate which spans are permitted to influence tool-calling decisions.

3. 3

   Layer with existing input/output filtering

   Combine static injection detection and content isolation techniques with dynamic, inference-time detection since static methods alone are shown to be insufficient against reasoning-time inducements.

4. 4

   Sandbox and validate tool outputs

   Treat all external tool and API responses as untrusted data; validate and sanitize before allowing them into the agent's reasoning context.

5. 5

   Monitor for anomalous tool invocation patterns

   Log and alert on tool calls whose triggering context originates from low-authority sources, especially when arguments deviate from expected user intent.