How to Not Waste Tokens (15-minute read)

When a process running an LLM agent is interrupted due to OOM (Out of Memory), a Deploy, or a forced system shutdown, the In-flight HTTP request that was already being generated is lost. This causes the problem of having to pay duplicate costs to regenerate output tokens that have already been paid for.

This is especially true for complex agent loops that perform multiple Tool Calls — if the process is cut off midway, the cost of all tokens generated in that turn is simply sunk. The higher the model's performance (e.g., GPT-5.5 vs GPT-5.5-mini), the more the cost burden of retrying grows exponentially.

To solve this, an architecture is proposed that places a Durable Buffer between the agent and the LLM provider. Deploying the buffer as a separate Worker or Durable Object offers the following benefits.

• Process independence: Even if the agent process terminates, the buffer process keeps running and receives the LLM's response to completion.
• Data persistence: Received data is immediately stored in SQLite, allowing a disconnected browser or a restarted agent to catch up.
• Cost optimization: Even if inference is interrupted, the already-generated tokens are safely recorded in the database, eliminating the need to call from scratch again.