The longer the conversation, the more it collapses

The key finding from building an open-source red teaming tool is that even agents that pass single-prompt tests easily break down in long multi-turn conversations.

In short conversations, the system prompt dominates, so refusals work well, but as conversations grow to around 50 turns, with over 40 helpful exchanges piling up, safety instructions effectively become a small fraction of the context. As a result, suddenly refusing after a long cooperative context looks unnatural to the model, and safety policies weaken.

The test method is phased escalation.

• First build trust through normal conversation
• Then probe boundaries with hypothetical questions
• Finally escalate with more direct requests at the end

Notably, this uses two conversation histories: one keeps only the clean conversation, while the other retains all attempts and scores. Approaches are retried with the agent's refusal records removed, so the agent loses the context of its previous refusal while the attacker can keep pressing from new angles.

The author states this approach was influenced by the Crescendo paper, OWASP LLM Top 10, and Meta GOAT, and emphasizes that such multi-turn attacks repeatedly break agents that had passed single-turn benchmarks.

The tool built on this is the open-source agent testing framework Scenario, with the repository at langwatch/scenario.