All articles
5 min readai / productivity

Adaptive attacks broke all 8 defenses: how to scope an AI agent

In the AI trainings I run, a team can wire an agent to their inbox, a CRM and a browser inside an afternoon. The demo lands well. The security question shows up two weeks later, and it has a specific shape: the hostile instruction never passes through the prompt box, it rides in on the data the agent reads. Qiusi Zhan's group took eight published defenses against that attack and got past all eight, holding a success rate above 50% 1.

What the research shows.

Kai Greshake's team described the class in 2023. They hid instructions at zero font size inside a web page, and a Bing Chat session reading that page turned into a social engineer that questioned its own user and passed the answers on. Their taxonomy sorts the damage into data theft, worming into documents the agent writes next, contamination of downstream systems, and unauthorized API calls 2.

Yi Liu's group pointed a black box attack called HouYi at 36 deployed LLM applications, products with paying users, and found 31 of them vulnerable 3.

Two benchmarks measure agents rather than chatbots. InjecAgent runs 1,054 test cases built from 17 user tools and 62 attacker tools. A ReAct-prompted GPT-4 carried out the attacker's instruction in 24% of cases. Reinforcing the injected text with a short hacking prompt lifted that to 47%, and ReAct-prompted Llama2-70B stayed above 80% in both settings 4.

AgentDojo puts 97 realistic tasks (an email client, an e-banking site, a travel booking flow) against 629 security test cases, and scores utility and security on the same run. The models solved under 66% of those tasks with nobody attacking them, and the benchmark's attacks beat the strongest agents in under 25% of cases 5. Read the two numbers together before handing an agent your calendar: it drops a third of the work it was hired for, and it takes orders from an outsider once in four or five tries.

Deployment figures point the same way. Anthropic measured a 23.6% attack success rate for Claude browsing autonomously in Chrome, then 11.2% after adding classifiers that screen incoming content and every proposed action 6. The figure comes from the vendor rather than an independent lab, and it is also the number they shipped with, alongside a rule that high-risk actions wait for user approval.

Filters buy time, they do not close the hole.

Zhan's 2025 paper is the one I quote most in the room. Eight defenses, each published with encouraging numbers. Attacks tuned against each defense in turn cleared all of them above 50% 1. Milad Nasr's team ran the same experiment at larger scale: 12 recent defenses spanning prompting, training and filtering, attacked with gradient descent, reinforcement learning, random search and human-guided search. Most fell above 90% 7.

A defense scored against the attacks published before it tells you about those attacks. Someone adapting to your specific filter is a different opponent, and the gap between the two numbers runs to tens of points.

The protocol.

  1. Answer three questions for each agent before you build it. Does it read untrusted content? Does it touch private data? Can it send anything outward? Three yeses put you one injection away from a breach. Cut one leg. An agent that reads the web and drafts into a document nobody auto-sends carries a different risk than the same agent holding mail access.
  2. Fix the tool list per task instead of letting the model pick from your full catalogue. The action-selector and plan-then-execute patterns 8 describe the shape: the plan comes from the user's request, formed before any untrusted text enters the context.
  3. Keep untrusted content out of the control flow. CaMeL extracts control and data flow from the trusted query alone, then enforces capability policies at each tool call. It solved 77% of AgentDojo tasks against 84% for the same undefended agent 9. Seven points of utility for a property you can verify.
  4. Gate irreversible actions behind a person: send, pay, publish, delete, grant access. The approval screen shows the recipient and the payload, not a summary of them.
  5. Log the provenance of every tool call, meaning which document, page or message sat in context when the agent decided to act. Without that field you cannot run an investigation.
  6. Build 20 test cases from your own corpus. Real emails, real tickets, with instructions planted where an attacker would put them: signature blocks, PDF metadata, white text on white, code comments, calendar invite descriptions. Run the set on every prompt change and every model upgrade. InjecAgent 4 and AgentDojo 5 are open source, and their attack templates transfer.
  7. Before you trust a defense, hand it to someone whose task is to tune an attack against it with the defense in plain view. Both adaptive-attack papers 17 measure what that person finds.

Start with step 1 on whatever agent you already have in production. If the answer comes back three yeses, the cheapest move this week is step 4: put the send action behind a click.

Sources.

  1. Zhan, Q., Fang, R., Panchal, H. S., & Kang, D. (2025). Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents. Findings of the Association for Computational Linguistics: NAACL 2025, 7116-7132. aclanthology.org
  2. Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz, M. (2023). Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security (AISec '23), 79-90. doi.org
  3. Liu, Y., Deng, G., Li, Y., Wang, K., Zhang, T., Liu, Y., Wang, H., Zheng, Y., & Liu, Y. (2023). Prompt Injection attack against LLM-integrated Applications. arXiv preprint. arxiv.org
  4. Zhan, Q., Liang, Z., Ying, Z., & Kang, D. (2024). InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents. Findings of the Association for Computational Linguistics: ACL 2024. aclanthology.org
  5. Debenedetti, E., Zhang, J., Balunović, M., Beurer-Kellner, L., Fischer, M., & Tramèr, F. (2024). AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. NeurIPS 2024 Datasets and Benchmarks Track. arxiv.org
  6. Anthropic (2025). Mitigating the risk of prompt injections in browser use. anthropic.com
  7. Nasr, M., Carlini, N., Sitawarin, C., Schulhoff, S., Hayes, J., Ilie, M., Pluto, J., Song, S., Chaudhari, H., Shumailov, I., Thakurta, A. G., Xiao, K. Y., Terzis, A., & Tramèr, F. (2025). The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt Injections. arXiv preprint, to appear at USENIX Security 2026. arxiv.org
  8. Beurer-Kellner, L., Buesser, B., Creţu, A.-M., Debenedetti, E., Dobos, D., Fabian, D., Fischer, M., Froelicher, D., Grosse, K., Naeff, D., Ozoani, E., Paverd, A., Tramèr, F., & Volhejn, V. (2025). Design Patterns for Securing LLM Agents against Prompt Injections. arXiv preprint. arxiv.org
  9. Debenedetti, E., Shumailov, I., Fan, T., Hayes, J., Carlini, N., Fabian, D., Kern, C., Shi, C., Terzis, A., & Tramèr, F. (2025). Defeating Prompt Injections by Design. arXiv preprint. arxiv.org

Keep reading