Vals Launches Independent Router Evaluation on Real-World Agentic Tasks

Vals is developing an independent, end-to-end evaluation for model routers, addressing the need to assess quality, cost, and latency on real professional work. The initiative stems from the observation that the AI frontier is jagged: on the Vals Index, five different models hold top spots across eight benchmarks, and no single model ranks in the top three on all of them. This fragmentation means relying on one model often compromises performance, while managing multiple models creates operational burdens.

The evaluation aims to measure how well routers capture the potential of model selection. Vals calculated that an oracle router, which picks the perfect model for every task in hindsight, would outperform the best single model by 14 points on the Vals Index while costing approximately 40% less. This highlights the significant efficiency gains possible through intelligent routing, which existing academic benchmarks—often based on saturated, isolated public questions—fail to capture.

Current router evaluations face limitations because they do not reflect the complexity of modern agentic workflows. Agents complete tasks over dozens of steps, where a routing decision affects subsequent context and costs. For instance, switching to a cheaper model might save money on one call but introduce errors that require additional calls to fix, or incur costs from losing prompt cache. Vals argues that routers must be evaluated on live, long-horizon tasks across domains like finance, law, and coding, rather than static prompts.

The new evaluation will run routers through the same harnesses used for the Vals Index, which includes benchmarks like Terminal-Bench 4.0, Excel Modeling, and Harvey’s Legal Agent Benchmark (HLAB). These tasks are expert-written, mostly private, and long-running, with some requiring hours of work. The evaluation will compare router performance against single-model baselines and the oracle reference, documenting each router's configuration and measuring task outcomes, cost per task, cache usage, and completion time. Vals plans to publish results on a public leaderboard soon.