CJ Olive Networks Builds Enterprise-Wide LLM Gateway Based on AWSome AI Gateway

Connected Amazon Bedrock, Claude Platform on AWS, and internal open models to a single endpoint to manage enterprise-wide LLM calls via a central gateway
Consistently applied a six-stage governance policy—including authentication, budget verification, model switching, call volume limits, and cost logging—to all requests
Prevented business interruptions by automatically switching to internal open models that do not deduct user Credits when the budget is exhausted, instead of blocking paid models
Implemented automatic scaling based on traffic and component-level fault isolation using a stateless architecture based on Amazon EKS and AWS Fargate
Simplified developer onboarding by automating VK issuance and Claude Code connection through a single internal SSO login via gateway-cli

Architecture and Operations

Separated storage for VKs and budget counters in Amazon ElastiCache for Valkey, usage history in Amazon Aurora PostgreSQL, and prompt content in Amazon S3
Unified source addresses with fixed private IPs using Private NAT Gateway and SNAT to handle IDC firewall requirements
Separated roles such as Gateway Proxy, Admin API, and Cost Worker to enable independent deployment by function and minimize the scope of impact during failures

Automatically handled internal SSO authentication, VK issuance, and Claude Code connection setup using the gateway-cli login command
Eliminated the burden of manual key management for developers by automatically reissuing VKs before expiration through the api-key-helper configuration
Ensured data security and traceability by storing prompt input and output logs with encryption and applying audit logs