Speeding Up Agentic Workflows with WebSockets on the Responses API

Codex's agent loop repeats file exploration, tool execution, and result integration, and as multiple Responses API round trips piled up, latency grew. In particular, back when GPT-5 and GPT-5.2 were running at around 65 TPS, model inference was slow enough that API overhead wasn't very visible, but on ultra-fast models like GPT-5.3-Codex-Spark, which target 1,000 TPS or more, CPU-side API processing costs became the bottleneck.

To reduce this, OpenAI first improved TTFT by about 45% through caching, reducing network hops, and improving the safety classifier. Even so, because the entire history of a long conversation was reprocessed on every request, the problem remained that each subsequent request repeated the same validation and state reconstruction.

The solution was a persistent connection. Initially, they considered WebSockets and gRPC bidirectional streaming, but chose WebSockets since it could be attached with almost no change to the existing Responses API's input/output shape. The core idea is to keep the connection alive, validating only the newly arrived input while keeping reusable state in memory.

The first prototype was more radical. It treated the agent's entire rollout as a single long Response: when a tool call was sampled, the API would pause asynchronously, send a response.done event, and then resume sampling once the client returned the tool result via response.append. This structure eliminated almost all duplicate work, but from a developer's perspective, the new interaction pattern felt too unfamiliar.

The version that actually shipped kept a familiar shape. It still uses response.create as before, but carries over the state of the previous response via previous_response_id. Within a WebSocket connection, the server keeps the previous response's state in a per-connection memory cache, and when a subsequent request arrives, it reuses that state directly instead of reconstructing the entire conversation history.

This cache includes the following:

• The previous response object
• Past input/output items
• Tool definitions and namespaces
• Reusable sampling artifacts such as already-rendered tokens

This structure enabled several optimizations.

• The safety classifier and request validator process only the new input, not the entire history
• Rendered tokens are accumulated and managed in memory, eliminating unnecessary tokenization
• Model resolution/routing logic is reused
• Non-blocking postinference work such as billing is overlapped with the next request

The results showed up immediately. After two months of work, they launched an alpha and deployed it first to a few coding-agent startups, and users reported up to 40% improvement in agentic workflows. Since then, Codex has moved most of its Responses API traffic to WebSocket mode, and on GPT-5.3-Codex-Spark it hit the target of 1,000 TPS, momentarily spiking as high as 4,000 TPS.

The ecosystem's reaction was similar.

• Vercel integrated WebSocket mode into the AI SDK, cutting latency by up to 40%.
• Cline's multi-file workflows got 39% faster.
• Cursor's OpenAI models got up to 30% faster.

WebSocket mode has become one of the most important new features since the Responses API launched, and it clearly demonstrates that as model inference gets faster, the surrounding services and systems need to speed up right along with it.