Online Learning for Cost-Efficient LLM Routing

Ramp operates an internal LLM gateway that processes trillions of tokens per day, and through dynamic routing across models and service tiers, it successfully cut AI costs by over 25% while also reducing error rates.

The starting point was the observation that OpenAI's flex tier doesn't always have higher latency than default. In fact, during business hours (9-5), flex tier latency spiked sharply, but outside those hours there was little difference from default. Since this pattern wasn't fixed, static rules couldn't handle it.

The core design principles are three:

• Reliability comes first — handling various failures such as provider outages and rate limits
• Latency matters differently depending on the purpose of the call (background automation vs. user-facing UI)
• Callers express their preferences by passing a model priority list to the gateway

To encode this into a single routing strategy, they adopted an approach that tracks real-time latency and failure distributions through online learning. Instead of fixed rules, it continuously incorporates actual measured data to make optimal routing decisions.