Finding End-to-End Latency Bottlenecks in Apache Flink Applications

To reduce end-to-end latency, you first need to break down the entire Flink application by operator and observe it. To do this, add two kinds of histogram metrics to each operator: processing time is the time it takes for map() or processElement() to process input and produce output, and non-processing time is all other time.

This breakdown immediately reveals the nature of the bottleneck. If processing time is large, you should check the application logic, blocking I/O, and Flink state usage patterns, while if non-processing time is large, you can suspect network I/O or execution of Flink's internal code. Flink itself has a LatencyMarker, but it causes significant performance degradation, doesn't allow separation of processing time from non-processing time, and its accuracy drops in complex jobs with timers, aggregation, and windowing.

Once you've found the bottleneck operator, dig deeper with an operator flame graph. A flame graph samples the call stack to visualize execution time per function, and can be checked in the Flink web UI. However, it's disabled by default, so you need to set rest.flamegraph.enabled: true in config.yaml, and by default sampling occurs every 3 minutes, causing performance degradation on that operator. For this reason, it's more appropriate to use it in a test environment with production-like load rather than on a live server.

The analysis results generally fall into three cases.

• If processing time is the bottleneck, the user code section takes up most of the flame graph. In this case, use code-level inspection to check for slow logic, blocking I/O, and state backend configuration.
• If non-processing time is high and the user code proportion is low, network I/O is likely the bottleneck. You'll need to tune taskmanager.network.* settings and check the network infrastructure.
• If non-processing time is high and Flink's internal code takes up a large proportion, the problem lies in internal paths such as serialization or state handling. One example is a case where the Kryo serializer became the bottleneck, where a response could involve changing the data class so that the faster POJO serializer is used instead of runtime reflection-based Kryo.

Finally, the most important point is that you need to re-verify the meaning of the metrics you're already collecting at the code level. In an actual case, only the processing time metric existed and there was no non-processing time metric, causing the Kryo bottleneck to be missed, and the definition of the existing metric also differed from what was expected, leading to significant trial and error. The starting point for diagnosing bottlenecks isn't more metrics, but first getting clear on exactly what time is being measured.