Everyone is building an LLM router, but we killed ours

Manifest deprecated the LLM router it launched in March as a core feature of its LLM gateway in June, and fully shut it down on September 1. This router classified requests into four complexity tiers—simple, standard, complex, and reasoning—and routed them to the appropriate model.

After operating it for about 4 months with 7,000 cloud users, the team concluded that the following problems outweighed the cost savings.

• It's hard to judge task complexity from the prompt alone. The actual difficulty only emerges in the context that follows—tool calls, web search, repository size, and so on. The same code improvement request has very different complexity on a personal website versus the Linux kernel.
• Caching is more effective than routing for cost savings. Cached input is 75-90% cheaper than uncached input, and applying prefix caching to system prompts and conversation history can significantly cut costs. For caching efficiency, it's advantageous to keep using the initially selected model, which weakens the effect of routing.
• Switching models undermines behavioral consistency. If the model changes mid-task, output quality becomes unstable, and it becomes harder for engineers to learn the characteristics and trade-offs of each model.
• The operational cost of unpredictability is high. In agentic workflows, evals, system prompts, observability, and more must be managed per model, so the maintenance cost can end up exceeding the savings gained from routing.

Manifest concluded that for most use cases, it's better to explicitly set the model, parameters, and prompt per request and consistently use a single validated model. However, they are not claiming that LLM routing is meaningless for all use cases.