GS Retail's Enterprise AI Gateway Operations: Budget Routing and Custom Guardrails

GS Retail shared its operations and governance case study for an enterprise AI Gateway built in collaboration with AWS. To support thousands of active users beyond a simple API proxy, they directly customized budget-based automatic routing, custom guardrails, and an observability pipeline.

Automated User Onboarding

To enable non-developer roles to easily install AI tools, the cloud infrastructure team created a .NET-based installation bundle. Users self-service issue Virtual Keys via the Install Center, and the installer automatically writes credentials to configuration files, eliminating manual copy errors.

Budget Management and Automatic Model Routing

To prevent runaway generative AI costs, cost limits were set by division, and a routing policy was applied to automatically switch from high-performance models to lightweight models when budget thresholds are reached. Leaders can transfer unused budgets to heavy users within the team budget pool to maintain business continuity.

Custom Guardrails Implementation

To address the per-call cost issue of Amazon Bedrock Guardrails, custom guardrails were implemented at the Gateway level. Utilizing LiteLLM Proxy's CustomLogger, PII detection and blocking before requests (Input Guard) and inspection after responses (Output Guard) are performed, applying sensitive information filtering to all calls at no cost.

Observability Pipeline and Data Storage

Audit logs were stored separately into distinct full text and metadata. Metadata is stored in Amazon DynamoDB to optimize query performance, while full prompts are stored in encrypted Amazon S3 to strengthen access controls. Additionally, the Leader Lounge dashboard allows executives to view AI usage status and costs by organization in real time.