Calming DynamoDB! Building a Real-Time Backpressure Architecture with Kafka and Redis

During early-morning hours, batch-style Kafka jobs were overusing DynamoDB, and even real-time APIs suffered throttling damage. The reason failures occurred even during low-traffic periods was that batch and real-time jobs were competing for the same DB resources.

Static controls like limiting consumer concurrency or using a RateLimiter were simple but clearly limited. When the DB was idle, resources were wasted, while a fixed value could actually worsen failures when real-time traffic suddenly spiked, making it necessary to have central control that adjusts itself based on DB state.

The solution was a real-time backpressure architecture. The structure is divided into three parts.

• Monitoring & Metrics: CloudWatch monitors the ratio of actual usage to DynamoDB's provisioned capacity
• Alert Propagation: When a threshold is exceeded, SNS calls a Webhook, and the message is fanned out to Redis Stream
• Control Service: A Spring application subscribed to alarms pauses or resumes the batch-style Kafka Consumer

The control metric was set as the ratio between DynamoDB's Provisioned throughput and actual throughput. Since CloudWatch's collection interval is 1 minute for read/write metrics but 5 minutes for provisioned metrics, the LAST function was used to correct for this interval difference.

Batch-style jobs were clearly distinguished using a dedicated annotation like @KafkaBackpressure, and Spring Kafka's KafkaListenerEndpointRegistry was used to manage the target containers. Since consumers can be rebalanced at runtime, the alarm state is re-checked in onPartitionsAssigned of ConsumerRebalanceListener so that newly assigned partitions are immediately paused as well.

As a result of controlling early-morning batch jobs with this approach over the past 6 months, the article states that read/write throughput decreased by 11% and 47.5% respectively. However, there are also limitations: CloudWatch's 1-minute aggregation can miss short spikes, and the current approach operates crudely like an on/off switch, so they are considering introducing more granular metrics and a Global RateLimiter in the future.