Kafka Streams k8s Migration: 80% Cost Reduction with KEDA and Consumer Lag-Based Scaling

Migrating the Log Transformer (a Kafka Streams application) from AWS EC2 to Kubernetes (k8s) resulted in over 80% infrastructure cost savings through a scaling design optimized for Kafka consumer workloads.

Limitations of CPU-Based HPA and Solutions

The existing EC2 environment relied on ASG cron schedules, making it difficult to respond immediately to traffic changes. Initially after adopting k8s, CPU-based HPA (Horizontal Pod Autoscaler) was applied, but due to the characteristics of Kafka's RangeAssignor, partitions were not evenly distributed among consumers. Consequently, even if some pods were heavily loaded, the average CPU did not exceed the threshold, preventing scaling from triggering.

KEDA and Consumer Lag-Based Scaling

To scale based on Consumer Lag, a metric that reflects actual load, KEDA (Kubernetes Event-Driven Autoscaling) was introduced. KEDA's Kafka scaler queries the total lag of a consumer group to adjust the number of replicas, allowing for appropriate responses regardless of partition distribution imbalance. Additionally, the scale-to-zero feature, which reduces the number of pods to zero during idle periods, eliminated idle costs.

Handling Midnight Traffic Spikes and Optimization

To handle traffic spikes caused by the midnight reset of daily missions, proactive scaling was applied using KEDA's cron trigger to pre-scale the number of pods. Furthermore, by multiplexing threads per pod, JVM overhead and rebalance events were reduced, minimizing processing latency.