Physical Prompt Injection Against VLM-Enabled Smart Glasses

Technical Analysis

The attack exploits the fact that VLMs treat scene text captured through a camera as a high-priority instruction channel, similar to a system or user prompt, rather than as untrusted visual data. An attacker plants adversarial text in the physical environment (e.g., on posters, screens, or objects) that is imperceptible or innocuous to humans but is parsed and executed as an instruction by the VLM's OCR/vision pathway. This crosses the boundary between physical world input and the model's instruction-following layer, giving the attacker indirect control over the model's summaries, decisions, or safety-critical outputs without any digital access to the device or its software stack. The paper identifies 6 threat vector categories and shows blind trust in embedded text over actual visual context, enabling goal hijacking of downstream agent behavior in AI-assisted decision-making tasks.

Detection Signatures

• Anomalous divergence between OCR-extracted text content and overall scene semantic context
• High-confidence instruction-like phrasing (imperative verbs, role reassignment cues) detected within scene text regions
• Sudden output behavior shifts correlated with presence of embedded text overlays in image input
• Text regions with unusual contrast/formatting inconsistent with natural signage (steganographic or adversarial patterning)
• Repeated flagged scene-text triggers across multiple sessions from same physical location

Remediation Steps

1. 1

   Deploy mask-based external filtering

   Apply pre-processing filters that detect and mask/redact suspicious embedded text regions in captured images before they reach the VLM inference pipeline.

2. 2

   Semantic-vector-based internal detection

   Integrate an internal detector that compares embedding-space semantics of detected scene text against overall image context to flag adversarial or out-of-context textual injections.

3. 3

   Separate instruction and perception channels

   Architect VLM pipelines to treat OCR-extracted scene text as untrusted data rather than as privileged instructions, requiring explicit user confirmation before acting on text-derived directives.

4. 4

   Adversarial robustness testing

   Red-team wearable VLM deployments against physical prompt injection using the paper's six threat vector categories prior to production release.

5. 5

   Output sanity-checking against visual context

   Implement cross-validation logic that flags outputs contradicting the broader non-textual visual context of the scene.