It Wasn't a Culture Problem: Airbnb's Alert Development Overhaul

Observability as Code (OaC) is a given assumption at large organizations, but the actual behavior of alerts could usually only be confirmed after deployment. As a result, teams repeatedly ran into a problem: either getting used to noisy alerts, or conversely missing important signals. Airbnb viewed this not as a culture problem but as a development workflow problem, and built a feedback loop that lets engineers preview and validate alert behavior before submitting a PR.

Airbnb's goal is for platform teams to ship validated monitoring with zero touch whenever they introduce components like Kubernetes, service frameworks, or databases. This system started 10 years ago as Interferon with 1,000 alerts and has now grown to 300,000 alerts, but as scale increased, the cost of validating changes grew sharply as well. Existing code review and unit tests alone couldn't answer questions like "how will this actually behave in production" or "what noise will this create," leaving developers dependent on side-by-side deployments and long-term observation.

The key to solving this problem was making alert behavior visible. Airbnb built a local-first workflow on top of Prometheus and introduced the following three stages in sequence:

• Stage 1: markdown diff shown in the CLI and CI, allowing alert changes to be reviewed instantly at the text level
• Stage 2: Change Report UI for visually comparing changes that will be reflected in production
• Stage 3: bulk backtesting, running alerts against historical data to see exactly when and why they would have fired

Backtesting in particular connects directly to Prometheus's rule manager, showing when and why a proposed alert would have fired against historical data. It backtests hundreds to thousands of alerts at once across an entire changeset, and surfaces quality signals like noisiness alongside the results to help prioritize review. When there are many changes, the table view can be sorted by noisiness so that alerts most likely to be problematic are reviewed first.

In building this system, Airbnb held to several principles. It prioritized compatibility, making full use of Prometheus's standard rule groups and query API as-is, and ran each backtest in a separate Kubernetes pod to avoid resource contention. It also put strong guardrails in place so that the backtesting system itself would never disturb production. Recording rule dependencies couldn't be fully automated, but the UI detects dependencies and prompts users to fix them first — shipping an 80% solution first rather than waiting for a complete one.

The results were clear. Alert changes are no longer something you "wait days to see after deployment" — they've become work that can be validated within a single PR. This let Airbnb migrate 300,000 alerts from a vendor to Prometheus, and cut the development cycle from weeks to minutes. Alert noise across the company dropped by 90%, and instead of a culture that tolerated noisy alerts, a culture of actively improving alert quality has taken hold.