Building an AI Gateway to Centralize Internal LLM Requests

InfraLab built an internal AI Gateway to centrally manage LLM API calls generated by over 30 services. The previous approach caused maintenance issues such as shortened model lifespans, increased validation burdens, rising costs, and distributed key management. Notably, as of June 2026, LLM costs accounted for 23% of AWS expenses.

Envoy AI Gateway Selection and Architecture

InfraLab ultimately adopted Envoy AI Gateway. Unlike LiteLLM Proxy or Portkey AI Gateway, it is a CNCF-managed project where core features (budgets, rate limits, authentication, etc.) are not locked behind commercial plans, allowing for flexible configuration of control mechanisms. Deployed alongside Envoy Gateway, it uses Kubernetes Custom Resources to declare routing and authentication policies, enabling GitOps-based infrastructure management.

Model Abstraction and Tiered Routing

An abstraction layer was applied that exposes only three tiers—low, medium, high—instead of hardcoding specific model names in service code. Each tier has a fixed maximum cost cap ($0.3, $1, $2) and intended use case, so the actual connected model can be changed in the gateway configuration without modifying service code. Priority is assigned to multiple provider paths for the same model to implement fallback; if a discounted path hits a rate limit, it automatically switches to the standard-price path.

Cost Reduction and Operational Results

After applying tiered routing, LLM costs in August 2026 decreased by 78% compared to the peak in June. This result stems from a combination of factors: the actual blended unit cost for the medium tier dropped to $0.18, which is one-fifth of the cap ($1); cache hit rates were maintained above 80%; and visibility into usage by service/feature prevented overuse. Connecting a new provider is handled with a single GitOps Pull Request, and model replacements are reflected across all services with just one change to the gateway configuration.