r/redteamsec • u/Itz_hany • 6m ago
I attacked a real AI agent 360 times with prompt injection. It resisted every attack, but the defenses broke normal tasks.
github.comI built **AgentArena**, an open-source red-team range for AI agents and MCP servers, and ran 600 live trials against Claude models.
**What it is**
- Fake agents with fake MCP servers (email, GitHub issues, notes, web). Everything is local and fake, and the "secret" is a random canary string, so nothing real can leak.
- An attacker LLM that writes prompt-injection and tool-poisoning payloads, sees why they failed, and evolves them.
- A judge (rules first, LLM only for fuzzy cases).
- An MCP gateway with 5 switchable defenses: tool-description scanner + hash pinning, taint tracking, per-task allowlist, spotlighting, and a monitor model.
- Results with 95% confidence ranges, so small samples don't pretend to be facts.
**Live results** (Claude Haiku 5.5 as the agent, Sonnet 5.5 as the attacker, 30 attempts per cell)
| | no defense | taint tracking | spotlighting | all defenses |
|---|---|---|---|---|
| Attack success | 0% (0–4%) | 0% | 0% | 0% |
| Legit "reply to the sender" task | 30/30 | 0/30 | 30/30 | 0/30 |
**What I found**
- The model resisted all 360 attack attempts with no defense at all, and often called them out ("this looks like a prompt injection, I ignored it").
- Taint tracking blocked a normal "reply to the sender" task every time, because the address came from the email and not from the user. Against this model it added no measurable protection, but it removed 100% of that task's usefulness.
- Spotlighting (marking untrusted text as data) cost nothing.
**Limits, to be upfront**
- The attacker model refused to write the attack in about 47% of its turns, so attack pressure was lower than a determined human's. "0%" means these attacks failed, not that the model can't be injected.
- Only one victim model, and only 3 of the 6 attack types were run live (all 6 run in the free mock mode).
- Total API cost was about $5.
There's a free mock mode (`arena demo`) that runs the full matrix with no API key.
GitHub: https://github.com/Ahany1440/agentarena
Full write-up: https://github.com/Ahany1440/agentarena/blob/main/docs/WRITEUP.md
I'd love feedback, especially attack ideas that would get past the model, or better defenses that don't break legitimate tasks.