Beyond Rate Limits: Scaling Access for Codex and Sora

As usage of Codex and Sora surged, the existing Rate limit approach hit limits that undermined user experience. Simply raising limits reduces fairness, while relying solely on usage-based billing (Usage-based billing) can cause delays or settlement errors.

To address this, OpenAI built a real-time hybrid access engine combining Rate limit and Credit. This system processes user requests through a 'Decision waterfall' model, designed so that users don't notice the system transition.

A user's request sequentially passes through the following layers:

• Rate limit application
• Free tier check
• Credit usage
• Promotion and Enterprise entitlement application

OpenAI built this system in-house to ensure real-time accuracy and transparency. This was because external platforms had limitations in instantly providing real-time balance checks and per-request details (whether allowed, amount consumed, limits applied, etc.).

The new system is based on a distributed usage and balance management system. Every request goes through a single evaluation path that synchronously checks Rate limits and verifies Credit when needed, while actual balance deductions are reliably performed through an asynchronous streaming processor.