Migrating a large-scale metrics pipeline from StatsD to OpenTelemetry / Prometheus

Airbnb moved away from its existing StatsD + Veneur + vendor storage setup toward OpenTelemetry Protocol (OTLP) and Prometheus-based storage.

First, they recommended OTLP for internal services, prioritized Prometheus for OSS workloads, and kept StatsD (DogStatsD) only for legacy paths. In this process, they used the OpenTelemetry Collector as an intermediate hub, and about 40% of services were changed to dual-write StatsD and OTLP simultaneously through a shared metrics library.

The effects of the transition were clear.

• CPU share for metrics processing (based on JVM profiling) dropped from under 10% to under 1%
• Improved stability with OTLP being more resilient to packet loss than UDP-based StatsD
• Eliminated the intermediate StatsD-to-OTLP conversion step
• More fully leveraged features like exponential histograms in the Prometheus environment

However, in extremely high-traffic services, enabling OTLP caused memory pressure, increased GC, and heap growth. The cause was extremely high-cardinality metrics and high publish volume, and for some services, delta temporality was applied to reduce the in-process state burden. This setting wasn't needed for all services and was used in a limited way only for services emitting an extreme number of samples.

For the aggregation stage, they chose vmagent instead of the existing Veneur approach. The reasons were as follows.

• Support for streaming aggregation
• Horizontal scalability through sharding
• Simple, easy-to-understand documentation
• A small codebase that's easy to modify and operate

The structure consisted of two stages: router / aggregator. The router shards using consistent hashing while excluding some labels, and the aggregator sums counters and maintains state. By leveraging the stable network IDs of Kubernetes StatefulSets, they used a simple approach where the router looks at a fixed list of aggregators, avoiding additional service discovery dependencies. This pipeline was internally customized to add native histogram support and Mimir-style multitenancy, and some changes were contributed upstream.

As a result, a single production cluster scaled to hundreds of aggregators and processed over 100 million samples per second. Costs were reduced roughly 10x, and having a centralized aggregation layer also brought operational flexibility, such as being able to block problematic instrumentation or temporarily re-export raw metrics when needed.

Finally, after the Prometheus transition, a sparse counter undercount problem surfaced. Counters that increment only once right after creation and then restart could lose their increment before the rate() calculation, and this was especially problematic for low-frequency but high-cardinality business metrics. Rather than fixing every call site individually, the solution was to apply zero injection at the aggregation layer. That is, when a counter is flushed for the first time, 0 is emitted first instead of the actual value, and the real value is emitted starting from the next flush, aligning with Prometheus counter semantics. This approach solved the undercount problem at the source while remaining invisible to users.

In summary, this piece is a real-world case study showing how Airbnb transitioned a StatsD-centric, large-scale metrics system to one based on OTLP + Prometheus + vmagent, and how they solved performance, cost, and accuracy issues along the way.