Amazon MSK Express Kafka Transaction Bottleneck Analysis and Optimization Guide

• Observed that throughput on a 3-node express.m7g.2xlarge cluster stalls at 64% of the network baseline when using Kafka transactions
• Empirically demonstrated that the bottleneck cause is not broker CPU or network, but rather coordinator round-trips and __transaction_state replication ack waits
• Adjustments must be made in the order of transaction batch size, producer count, and cluster scaling; the efficiency of scale out versus scale up varies depending on batch size
• If doubling the number of producers does not increase throughput but only increases commit latency, it can be identified as a coordinator limit

Bottleneck Diagnosis and Causes

• Fixed costs incurred in the commit path occur regardless of transaction batch size, determining the throughput ceiling
• 85–90% of the processing time for EndTxn and WriteTxnMarkers was measured as replication ack wait (RemoteTimeMs)
• Explains why throughput stagnates despite low broker CPU usage at 44% and the absence of network throttling

Optimization Strategies

• Increasing transaction batch size to distribute fixed commit costs offers the highest cost-effectiveness
• If throughput does not rise when increasing the number of producers, it indicates a coordinator limit; therefore, adjust batch size before considering scaling
• When scaling out, __transaction_state partitions must be distributed across all brokers to prevent up to 71% performance loss in high-load scenarios
• Initial sizing must be performed based on the minimum value among network limits, coordinator limits, and client limits