Pay Attention to the Gaps

The success or failure of production AI depends less on model scores than on the execution system. AI21 CTO Barak Lenz believes an AI Operating System (AI OS) must handle validation, routing, orchestration, and decomposition all at once.

• The Validation Gap is the difference between Success@1 and Success@N. The Oracle approach—running multiple attempts and picking the best result—boosts performance, but without a verifier that gain can't be recovered. In AI21's multi-scale indexing experiments, an additional 20-40% of headroom appeared compared to traditional single-path RAG.

• The Contextualization Gap means that the model that's best on average differs from the model that's optimal for a specific input. Since easier problems can be handled by smaller models, a Portfolio of Actions is needed, where an Action Model first determines the path and cost for each input.

• The Latency Gap shows that cost reduction and time reduction are different constraints. To reduce time, multiple paths must be run in parallel, and as soon as one is verified, the rest must be terminated immediately. This requires structured plans that separate orchestration from LLM inference, and the fastest successful path can be 3x faster than the average.

• The Decomposition Gap points to the limits of conventional agents that split a task once and then push through sequentially. Complex, long-horizon tasks require fractal decomposition that decides at every step whether to branch, stop, or reroute, and AI21's Maestro incorporates such branching points, producing better results than a ReAct loop.

Ultimately, useful production AI resembles not a single LLM but an AI OS that understands resources and creates, stops, and reallocates tasks. The recent GPT-5.3 Codex case is also cited as a sign pointing in this direction, with the model involved in its own development, debugging, and deployment.