OpenAI Habitat scales to 1 billion users, cutting global read latency from 120ms to 10ms

• Global read p50 latency reduced from approximately 120ms to 10ms, a 12x improvement, following performance optimizations initiated in early 2026.
• Habitat serves over 1 billion users across 40 geographic regions with millisecond-level responses, processing more than 500PB of data on Azure Cosmos DB.
• Multi-tenancy strategy evolved from isolated containers to shared containers with hot-key throttling and Quality of Service (QoS) prioritization, eliminating provisioning and noisy neighbor incidents.
• Reliability improvements included resolving failover stampedes and optimizing the control plane, supporting a single account with tens of PB of data and over 20M QPS.

Performance Optimization

• Read requests constitute over 90% of Habitat traffic, driving a focus on latency reduction.
• Data residency requirements and physical cloud capacity limits (power, cooling, storage) prevent placing all data near every user.
• Small, latency-sensitive datasets like authentication metadata are replicated across multiple Azure regions, with some accounts replicated to all 40 regions and others to at least one region per continent.
• A transparent data movement and routing layer was built to handle datasets with different replication topologies.

Caching and Infrastructure

• Integrated distributed caching using Valkey servers placed near user request infrastructure to mask cross-region latency.
• Change Data Capture (CDC) based invalidation was adopted, referencing Meta's TAO system, to maintain low data staleness while preserving high hit rates.
• Habitat service code was rewritten from Python to Rust to reduce proxy overhead and further lower latency.
• CDC infrastructure now handles millions of captured changes per second across thousands of containers and over 100 Kafka topics.

Future Scaling

• OpenAI is building an exabyte-scale online storage system optimized for large Habitat data types.
• The new system combines Rockset's RocksDB Cloud core with Raft-based replication and implements the same document model as Azure Cosmos DB.
• Disaggregated storage and physical replication are used to reduce flash capacity requirements, though peak QPS is lower than the current system.