AI BriefingTwo Flink Autoscaler Stories
Key point
Netflix is transitioning to an open-source solution from the Apache Flink community to overcome the limitations of its in-house autoscaler.
Details
Netflix has been using Apache Flink for stream processing since 2017 and, as of 2026, operates over 30,000 Flink jobs across multiple AWS regions. Most jobs are automatically generated by the management platform Data Mesh, while some consist of complex stateful pipelines for personalization or advertising.
The first in-house autoscaler monitored cluster metrics externally, achieving a 25–45% reduction in resource usage across thousands of managed pipelines. However, this system could only adjust a single knob (the total number of TaskManagers), which limited its effectiveness for jobs with complex multi-operator DAG structures. Additionally, issues with the accuracy of external metrics and maintenance costs became burdens.
Consequently, Netflix adopted an open-source autoscaler developed by the Apache Flink community. This system estimates the actual throughput (TPR) of each operator within the job to calculate parallelism across the entire graph, allowing for efficient management of complex workloads that the previous system could not handle. Netflix currently operates both systems and is gradually converging toward the open-source version.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Google Includes Premium Colab Benefits in AI Subscription TiersPirate Face Launched: Decentralized Infrastructure to Protect Hugging Face Models from Deletion