AI Gateway Sorts Providers by Cost, Latency, and Throughput

By default, AI Gateway determines order by jointly reflecting provider reliability, output quality, cost, and response speed, but now you can directly specify the priority criterion with sort. Put cost, ttft, or tps into providerOptions.gateway.sort.

• cost: lowest listed price per 1 million input tokens first
• ttft: lowest median time to first token first
• tps: highest median tokens per second first

Since rankings are calculated at request time, newly added providers, price changes, and changes in observed latency and throughput are automatically reflected without any code changes. For models with more than 5 providers and large cost differences, like GPT OSS 120B, you can use sort: 'cost' to try the cheapest provider first.

Providers are tried in the sorted order, and only fall back to the next provider when the top provider is unavailable. It also works together with zeroDataRetention, so for example with Deepseek V4 Pro you can first select only ZDR providers and then re-sort them by ttft. When used together with order, the specified providers are placed first, followed by the rest arranged according to the sort criterion.

The sort block in the response metadata contains the providers considered, the metric values used, the actual order attempted, and even providers pushed back due to degraded health.