Model routing is simple. Until it isn't

Model routing looks like a simple classification problem. It seems like sending simple requests to a low-cost model and difficult tasks to a high-performance model would reduce costs while maintaining performance. But in reality, that's not the case.

Problem 1: Cost is more than model pricing

GPT-4.1 should be cheaper than Claude Sonnet, but in practice the opposite was true. When processing 417 tasks in the AppWorld Test Challenge, Sonnet cost a total of $79 ($0.19/task) while GPT-4.1 cost $155 ($0.37/task). Looking at token pricing alone, GPT-4.1 should have the advantage, but prompt caching determined the overall cost. Agentic workloads reuse most of the context at each step, and Sonnet's lower cache-read pricing offset its higher base pricing and longer processing paths.

Problem 2: Complexity is more than task difficulty

At routing time, the actual task difficulty is unknown. "Summarize a contract" seems simple but may require search, regulation checks, tool use, and iterative refinement. Conversely, a technical prompt could be handled efficiently by a small specialized model. You can't predict the actual difficulty until you execute the task.

In addition, production environments must simultaneously balance cost, latency, model expertise, and reliability. Enterprise deployments add further constraints such as regulatory compliance, data residency, privacy protection, and approved model lists.

Problem 3: Latency is more than model speed

The intuition that bigger models are slower and smaller models are faster only holds partially. Routing itself adds overhead, and infrastructure factors such as the hardware environment, cache state, and endpoint load determine the overall response time. Routing at every step increases flexibility but also increases latency and operational complexity.

Shift to an optimization problem

Based on these lessons, the router that was developed approaches routing not as a classification problem but as an optimization problem. Instead of asking "which model is best," it simultaneously optimizes cost, quality, and latency, while keeping the router itself lightweight so it doesn't become a bottleneck (about 6ms and 2KB of memory per task).

The AppWorld Test Challenge results demonstrate the superiority of this approach. The latency-optimized configuration (Configuration 1) achieved 84% accuracy, $93, and 83 seconds, delivering a 21% cost reduction and 9% latency reduction compared to running Opus alone, with only a 4% drop in accuracy. The comparison target, a difficulty-based router (teal diamond), showed higher costs within a similar accuracy range.