In-house LLM on Amazon Bedrock: From Key Issuance to Cost Blocking: F&F's LiteLLM Gateway Case Study

To address API key management and sudden cost escalation issues arising from the adoption of generative AI within the organization, F&F built its own LLM platform. In particular, real-time quota limits and monitoring features were key to preventing unexpected billing from the use of high-cost models.

The platform leverages the stable model serving capabilities of Amazon Bedrock, while introducing a LiteLLM gateway to supplement insufficient operational requirements. The overall architecture consists of two areas as follows:

• Control Plane (Cloud Portal "LLM Lite"): Handles the operations console, including key issuance, model approval workflows, budget settings, and usage dashboards.
• Data Plane (LiteLLM Proxy on Amazon EKS): Acts as the single gateway for all LLM calls, performing model routing, access control, budget enforcement, and usage aggregation.

User identification utilizes SSO based on Microsoft Entra ID. Personal usage is managed by email, while service usage is managed by svc:{service_name} identifiers, designed to prevent individual experimental usage from mixing with constant service calls. This provides members with an optimized experience, completing everything from issuance to the first call in just 3 minutes.