Monitoring reliably at large scale

Airbnb pointed out the problem that if the observability stack sits on the same foundation as the systems it monitors, dashboards and alerts go down right when an outage happens. The core risk was a circular dependency where the metrics pipeline depended on the very infrastructure it was supposed to observe.

The solution was to give each internal customer a redundant, highly available path. However, they didn't take on responsibility for redundancy across the entire external failure domain, and clearly scoped their role to reliably delivering observability signals to internal customers.

At the compute layer, they moved to a dedicated Kubernetes cluster separate from product/infrastructure applications. The cloud team continued to handle operations and maintenance, reducing operational burden, and changes were validated one at a time on lower-priority clusters first to reduce risk.

At the network layer, they moved away from the Istio service mesh and built a custom Envoy-based L7 ingress. This layer routes read/write requests to the appropriate backend, and distributes over 1,000 services across a single global user space using per-service tenant headers.

• Separating observability traffic from business traffic reduced conflicts.
• It became possible to add custom features like metrics mirroring and fine-grained access control.
• Even if the shared data plane is blocked, the observability path remains independent.

Finally, for meta-monitoring, they ran separate Prometheus and Alertmanager instances, each as an HA set. These are placed on different nodes and AZs from the observability stack, and alerts flow through AWS SNS, after which a CloudWatch alarm monitors the rate of incoming messages. If the signal stops, a Dead Man's Switch triggers, immediately paging on-call.

The key point is treating monitoring itself as just another production system, and ensuring an independent path that survives even during outages.