OpenAI Deploys Pando Git Backend, Cutting Median Push Latency by 80% and Handling 1PB Daily

• OpenAI built and deployed Pando, a custom Git backend for its monorepo, reducing median push latency by nearly 5x (or nearly 80%) and delivering nearly a petabyte of Git data per day.
• Pando replaces traditional three-phase commit and write-ahead log approaches with a causal consistency model, allowing O(1) metadata writes for pushes and O(log Δ) reads for fetches.
• The system reuses stored packfiles instead of generating client-specific ones, keeping median packfile sizes under 20 KiB and 99th percentile under 3 MiB in the OpenAI monorepo.
• Pando was developed in 10 weeks using Codex, with GPT-6 Astra used to write Lean and TLA+ for formal verification, and has served production traffic since April.

Architecture and Consistency

• Adopts causal consistency to guarantee serializability for individual ref updates without requiring point-in-time snapshots of the entire repository.
• Pushes are stored as immutable objects in a blob store with ref versions recorded in a metadata store, ensuring read-your-own-write consistency.
• Fetches reconstruct necessary ref versions to select specific packfiles, leveraging the Git protocol's packfile-uri feature for inline empty pack transmission.

Performance and Scaling

• Delivers nearly a petabyte of Git data per day, with over 99% of requests served from local disk caches via the Express service.
• Uses consistent hashing to route packfile requests across multiple geographic clusters, maintaining high cache hit rates.
• Scales horizontally by adding servers, metadata stores, and blob stores in parallel, saturating 10 Gbps network links on standard cloud VMs.

Implementation and Maintenance

• Implemented in Rust with conformance test suites and deterministic simulation harnesses to verify invariants and race conditions.
• Uses a log-structured merge-tree design for compaction, collecting L0 packfiles into geometrically expanding covers up to L4 to improve compression and selection speed.
• Contributes improvements to the upstream Git project, including parallel download support, explicit haves, and byte count tracing for packfile-uri.