Maestro Deep Research Agent

AI21 Maestro is a framework for jointly optimizing agent quality, cost, and latency. In this experiment, it automatically swept a massive search space combining model selection, prompts, tool configuration, and execution policies, targeting BrowseComp-Plus and Deep Research Bench 1.

BrowseComp-Plus verifies retrieval precision and synthesis in deep corpus search, while Deep Research Bench 1 verifies long-form report generation and quality. Maestro broadly explored the possible operating points on these two benchmarks, revealing a Pareto frontier that is hard to see through manual tuning.

There were three key axes.

• Model and agent configuration: Compared BM25 sparse retrieval, Qwen2-7b dense retrieval, Reason-ModernColBERT late-interaction retrieval, and full-document retrieval tools.
• Best-of-N scaling: Boosted performance by running the same configuration multiple times to exploit run-to-run variance and selecting the best candidate.
• Ensemble scaling: Combined complementary configurations by leveraging the fact that different models and retrievers have different failure patterns.

In particular, when properly calibrated, the model's self-generated confidence score could be used as a runtime verification signal. Selecting candidates using this signal allowed near-oracle selection to be reproduced in actual operation. For example, an ensemble combining Minimax + LateInteraction run 8 times, GPT-5 + LateInteraction run 2 times, and GPT-5 + Dense run 2 times achieved higher accuracy at lower cost and latency than a single configuration of GPT-5 + LateInteraction run 4 times.

The message is ultimately clear. Agent optimization is not a matter of picking a single model, but a compound search problem combining models, tools, scaling, and execution policies, and Maestro automates this search to find production-viable operating points.