The data is precise, and it is uncomfortable. SADF's research, presented at DEF CON 34's AI Village, holds Claude Sonnet constant and measures Attack Completion Rate (ACR) across five orchestration targets. Direct API: 15.5%. CrewAI: 11.9%. LangChain: 18.1%. AutoGen: 20.0%. SmolAgents: 31.1%.
Same model. Same five attack goals. A 2.6x security spread. The vulnerability is not in the weights. It is in the glue.
For anyone who spent 2017 dissecting Uniswap v1's transferFrom logic, this pattern is structurally familiar. We learned then that the execution layer dictates the security outcome — a gas meter can be as dangerous as a flawed invariant. Tracing the SADF attack surface delta back to the orchestration layer is the same analytical move, applied to Agent systems.
SADF's experimental design is controlled attribution. Fix the model, vary the framework, isolate the incremental attack surface. The direct API serves as the control baseline; CrewAI, LangChain, AutoGen, and SmolAgents are the treatment arms. Any ACR difference above the 15.5% baseline belongs to the framework layer, not the model. The attribution logic is clean. 5,119 evaluation rows. 32 payloads. Eight failure modes codified: Tool Call Hijacking, Output Poisoning, Cross-Tool Injection, Memory Poisoning, RAG Poisoning, Delegated Authority Abuse, Multi-Agent Propagation, Context Boundary Violation.
The scale: 5,119 evaluation rows execute against a simulated tool environment that touches no real systems or credentials. The containment is deliberate — ethically sound — but it is also a limitation. More on that later.
This taxonomy is the study's quiet foundational contribution. Before SADF, Agent security discussions collapsed into a vague "jailbreak" bucket. Now the industry has a shared vocabulary for describing which component failed and where. Comparative work becomes possible. That alone is worth the price of admission.
The methodological highlight, though, is the scoring correction. The team discovered that naive substring matching overestimated Claude's vulnerability by a factor of four to six. They did not bury the flaw. They built a refusal-filtered scoring mechanism, recalculated, and published: Claude Sonnet's true ACR at 15.5%; Claude Haiku at 22.3%. In a field where inflated benchmark claims are the norm — and where the crypto audit industry has its own history of harnesses that do not match production — watching a team audit its own instrument and publish the correction is the rarest form of scientific integrity. My fraud proof work on Optimism's dispute windows taught the same lesson. Seven days felt adequate against naive state-root lies. Simulating malicious submissions exposed edge cases the naive threat model never considered. The instrument determines the conclusion.
The framework-level data is where the real insight lives. CrewAI's discrete task isolation architecture achieved 11.9% ACR — the lowest of all five targets, below the raw API baseline. Its separation of concerns suppresses cross-task contamination. SmolAgents, at the other extreme, posted 31.1%, with two distinctive failure signatures: a unique 20% RAG Poisoning success rate and a 64% Context Boundary Violation rate. The framework's memory-sharing and tool-routing design determines how much context crosses trust boundaries. That is an architectural variable, fully decoupled from model intelligence.
Note what the decomposition exposes. Tool Call Hijacking and Output Poisoning appear across all frameworks — they are framework-agnostic. But RAG Poisoning at 20% and Context Boundary Violation at 64% concentrate where the framework shares memory and context liberally across agents. The failure mode taxonomy is not decorative. It maps directly to architectural decisions: shared memory, tool-routing granularity, delegation policy. That mapping is the research's real payload.
This is not classroom theory. The CVE registry already carries framework-level vulnerabilities — CVE-2026-62830 in Azure's SRE Agent, CVE-2026-9198 in Langflow. Production systems are being exploited through orchestration defects today. The experimental evidence and the incident evidence converge: model safety alignment is insufficient because the model is only one link in the execution chain. Based on my own audit experience — particularly the ERC-721A integer overflow I flagged before mainnet — the distance between theoretical framework risk and exploited framework risk is mostly a matter of disclosure timing. SADF names the liability in advance. That is rare, and it is valuable.
The procurement implication follows directly. Enterprise Agent selection currently centers on "which LLM is smartest." SADF converts that into a rational RFP question: which framework minimizes ACR for equivalent model quality? A 2.6x gap between frameworks is procurement-grade data, not noise. The most probable commercial transmission is Security-Evaluation-as-a-Service — safety firms converting this methodology into repeatable Agent audits embedded in CI/CD pipelines. Recurring revenue beats one-off consulting.
The contrarian read is where this gets honestly uncomfortable.
Simulation removes two production variables: permission-boundary granularity and real tool response timing. Cross-tool injection is acutely sensitive to timing and state mutation. In a simulated environment, attackers never race a real concurrent process. The real-world deformation of these failure modes is likely understated.
The payload corpus is thin. Thirty-two payloads, likely researcher-selected, may not represent real attacker distribution. Rare but headline-grade vectors may be structurally missing from the sample. The 4-6x scoring inflation was corrected; the payload distribution skew remains unquantified.
The model×framework interaction is the largest unresolved variable. AutoGen at 20.0% with Claude says nothing about AutoGen with GPT-5.4, DeepSeek, or Llama. The ranking may not survive a model swap. The model×framework interaction effect is a regression risk embedded in every procurement decision made from this data.
There is also a gap between the claim and the data. The study says it covers eight architectures yet presents detailed ACR numbers for only five. The remaining three may have incomplete data or non-comparable configurations. Declared coverage and actual coverage are not identical.
The transparency signals cut both ways. The SUPERSEDED folder — an earlier version claiming ten coordinated architecture disclosures — suggests early-phase instability. Honest, but a warning: conclusions must pin exclusively to the current version. Meanwhile, the publication channel is misaligned. A blockchain/Web3 outlet reaches retail crypto readers, not the enterprise CISOs and Agent developers who could act on this data. The research's commercial transmission value is high. The channel is structurally wrong for it.
SADF is a proof of concept, not a standardized evaluation suite. The direction is still correct. Agent security evaluation must move from the model unit to the model + framework + tool environment triad.
The 2.6x ACR gap is the signal. Architectural choice now matters more than model alignment. The first major cross-tool production compromise will not be a model failure. It will be an orchestration failure. The question is whether procurement teams adopt ACR-based evaluation before — or after — that CVE makes the headlines.
