AI21 Proposes Verifiability Litmus Test to Guide Agent Architecture

AI21 researchers propose a verifiability litmus test to guide agent architecture before scaling compute. The core question is whether a candidate answer can be checked against explicit conditions. If yes, the architecture should prioritize verification and selection; if no, it should rely on diversity and aggregation. This approach aims to prevent inefficient budget allocation when task structures require different strategies.

Agentic Search: Verification Wins

In agentic search tasks where the answer is a single entity with explicit conditions, verification is efficient. AI21 demonstrated that using an independent verifier rather than majority voting significantly improves accuracy. By training an 8B verifier, they matched the quality of a frontier verifier at 3.2x lower cost. On an all-open-source ensemble, this approach lifted accuracy by 28% over plain majority voting, achieving more than 80% of a frontier verifier's performance for roughly 1/100th the price.

Deep Research: Diversity and Merging

Deep research tasks lack a pre-defined set of correct facts, making completeness unverifiable. Oracle experiments showed that selecting the single best report hits a ceiling below state-of-the-art performance. However, merging reports from multiple agents clears this ceiling. By combining outputs from seven low-ranked agents, AI21 produced a merged report that took first place on DeepResearch Bench II, surpassing both the individual agents and the previous state of the art.

RAG Indexing and Coding: Mixed Verifiability

RAG indexing presents a unique challenge because chunk size must be decided before queries are known. Committing to a single chunk size can leave 20-40% of recall on the table, necessitating multi-resolution indexing despite higher costs. In agentic coding, verification is binary at the end (tests pass/fail) but unverifiable during the process. AI21 used a tiered approach: junior models (MiniMax-M3) explore the repository, a senior model (GPT-5.2) distills findings, and a frontier model writes the patch. This pipeline achieved an 80.8% resolve rate on SWE-Bench Pro at $5.99 per task, compared to $18.28 for a solo frontier agent.