Repeat-After-Me: Adaptive Visual Prompt Injection Against VLM Agents

Technical Analysis

The attack embeds an adversarially optimized visual pattern into an image that, when processed by a VLM-based agent, causes the model to emit attacker-chosen output — including precisely formatted, parseable native tool calls with correct function names and arguments — despite no textual or verbal authorization from the benign user prompt. The entry point is any untrusted image ingested by the agent (e.g., shared in a Discord channel), making this a supply-chain-adjacent, model-input attack rather than a text-channel injection. Optimized perturbations transfer across surrogate and commercial models (43-66% ASR retention) and achieve >80% ASR on open-weight models and >47% on commercial frontier models like GPT-5.5, indicating the technique generalizes rather than being model-specific. In the demonstrated OpenClaw case, the injected tool call overwrites a configuration/instruction file (TOOLS.md), which crosses the boundary from a single malicious perception event into persistent alteration of the agent's future behavior — enabling downstream RCE and credential/secret exfiltration by later, unrelated agent turns.

Affected Systems

OpenClaw

Detection Signatures

• Unexpected tool-call generation immediately following ingestion of an image with no corresponding textual instruction from the user
• Modifications to configuration/instruction files (e.g., TOOLS.md) triggered by non-text/image-derived agent turns
• Tool calls whose arguments do not semantically match the accompanying user message context
• Repeated near-identical tool invocations shortly after image uploads from external/untrusted users
• Anomalous high-confidence structured output (JSON/function-call syntax) attributed to vision input rather than text input

Remediation Steps

1. 1

   Isolate image-derived actions from privileged tool execution

   Require explicit, separately-verified textual user authorization before executing any tool call whose triggering context includes untrusted image content.

2. 2

   Protect configuration/instruction files from agent self-modification

   Make files like TOOLS.md read-only to the agent at runtime, or require human review/approval for any write to agent behavior-configuration files.

3. 3

   Input sanitization and adversarial image detection

   Deploy pre-processing filters (e.g., JPEG re-compression, resizing, noise injection) and adversarial-perturbation detectors on untrusted images before VLM ingestion.

4. 4

   Least-privilege tool-call gating

   Enforce allow-lists and schema validation on tool calls, with anomaly detection for calls not clearly justified by the visible conversation context.

5. 5

   Segregate untrusted multimodal input channels

   Treat images from untrusted/public sources (e.g., Discord users) as untrusted data streams, sandboxing any resulting agent actions and requiring human-in-the-loop confirmation for sensitive operations.

6. 6

   Monitor transferability risk across model deployments

   Assume attacks optimized against one model may partially transfer to others in use; test defenses against surrogate-optimized and cross-model transfer attacks, not just white-box cases.