Self-Managed Agentic AI Platform Operated on Amazon EKS: Achieving Operational Stability Through Infrastructure Automation and Observability

As LLM-based agents move to production, the cost, latency, hallucination, and data governance problems of relying on a single massive model grow larger. A heterogeneous multi-model architecture is needed, where high-performance LLMs handle complex reasoning while domain-specific SLMs handle repetitive tasks like FAQs, classification, and extraction—and it's estimated that 40–70% of agent LLM calls can be replaced with fine-tuned SLMs. The Kubernetes ecosystem is also expanding AI-native capabilities through Gateway API Inference Extension, Kueue, LeaderWorkerSet, and more.

The platform automates GPU infrastructure on top of Amazon EKS Auto Mode, and uses the Bifrost AI Gateway to unify self-hosted vLLM and Amazon Bedrock into a single OpenAI-compatible endpoint. Two-layer observability separates and reports on infrastructure costs and agent bottlenecks independently, and a hybrid approach—starting with Bedrock and scaling to self-hosted EKS as needed—is a practical strategy.

The core components are as follows.

• EKS Auto Mode automatically manages VPC CNI, EBS CSI Driver, and CoreDNS, and even includes GPU drivers, Container Toolkit, and Device Plugin in the AMI, so GPUs can be used just by requesting nvidia.com/gpu: 1.
• The GPU NodePool uses both Spot and on-demand instances, and reduces idle GPU costs through WhenEmptyOrUnderutilized and 30s consolidation. For components where automatic ARM64 selection must be avoided, kubernetes.io/arch: amd64 should be explicitly specified.
• Bifrost routes using the provider/model format, and even vLLM on internal DNS can be attached just like an external API. For the Bedrock provider, an AWS credentials Secret is added, then registered via the Web UI or /api/providers. Models for different query types are separated using ConfigMap values like MODEL_SIMPLE and MODEL_COMPLEX, allowing switching without code changes.
• vLLM is deployed with the Qwen3-8B example using nvidia.com/gpu: 1 and --max-model-len 4096 settings. On EKS Auto Mode, to avoid the possibility of Graviton being selected, a direct Deployment is used instead of the vLLM Production Stack, placing it on the GPU NodePool.
• Langfuse is deployed via Helm chart, using the OpenTelemetry-based SDK with @observe and propagate_attributes(). It includes ClickHouse, PostgreSQL, Redis, Zookeeper, and S3 (MinIO), and on EKS Auto Mode, the ebs-auto StorageClass must be set up beforehand. Without keys, it operates as a no-op, allowing separation of development and production environments.

In the customer support demo, LangGraph classifies queries, routing simple inquiries to self-hosted-vllm/qwen3-8b and complex inquiries to bedrock/global.anthropic.claude-haiku-4-5-20251001-v1:0. The Bifrost dashboard shows request volume, tokens, and latency per provider, while Langfuse allows checking traces and evaluation results for each agent, enabling simultaneous routing and quality tuning.