AgentProv: Action-Channel Auditing of Agentic LLM API Backbone Substitution
First seen Sep 2, 2026 · Updated Sep 2, 2026
This is an academic research paper describing a defensive auditing technique, not an active threat or exploit. AgentProv helps detect when commercial LLM API providers silently swap, quantize, or wrap the advertised model, using tool-call patterns instead of unreliable text-output analysis. It is a beneficial transparency/integrity tool for consumers of agentic LLM APIs, not an attack vector.
Technical Analysis
The paper addresses a real supply-chain integrity concern: LLM API providers may substitute the advertised backbone model for cost reasons without disclosure, and prior audit methods relying on text-output distributions are unreliable for agentic APIs because tool-calling stacks (OpenAI, Anthropic, Gemini, Cloudflare Workers AI, LangGraph) often discard text entirely and only expose structured tool-call actions. AgentProv instead fingerprints models via their categorical tool-call distribution and applies an MMD permutation test to detect substitution, achieving 100% detection on 630 checkpoint pairs while reducing false positives caused by provider-injected system prompts (7% vs 53-67% for prior text-based methods). The 'attacker' in this scenario is a dishonest API provider silently substituting models; the defenders are downstream developers/auditors who need assurance the model they are paying for and building agents on top of is the one actually served. There is no described exploit against agent frameworks, no prompt injection, no tool poisoning, and no cross-agent boundary crossing — this is a verification/audit methodology.
Affected Systems
LangGraph, OpenAI API, Anthropic API, Gemini API, Cloudflare Workers AI
Detection Signatures
- N/A - this is a defensive audit tool, not an attack pattern. Relevant indicators for providers/consumers to watch: unexpected shifts in tool-call category distributions across API calls to the same claimed model; discrepancies between token-count side-channels and expected system-prompt-free baselines; MMD permutation test results flagging distributional drift between claimed and served model checkpoints.
Remediation Steps
- 1
Adopt action-based provenance audits
Organizations relying on third-party agentic LLM APIs should incorporate tool-call distribution audits (e.g., AgentProv-style MMD tests) rather than relying solely on text-output comparisons to verify backbone model identity.
- 2
Contractually require model transparency
Enterprises should require API providers to disclose any model substitution, quantization, or wrapping in SLAs, and periodically audit compliance using independent action-channel or side-channel techniques.
- 3
Monitor token-count and system-prompt side-channels
Use token-count and injected system-prompt detection as a complementary signal to distinguish legitimate deployment variance from actual model substitution.
- 4
Version-pin and checksum where possible
Where APIs support it, pin specific model checkpoint versions and monitor for silent version drift over time.
Industries Most Exposed
Respond to this threat
Pro subscribers get a full AI-generated incident-response playbook for this threat — detection, containment, eradication, and recovery steps — plus an unlimited AI Threat Advisor for questions about your environment.