Why harnesses matter for energy (and token) efficiency

Prompt Caching as the Primary Efficiency Driver

Neuralwatt CEO Chad Gibson analyzed cloud traffic from September 8 to October 7, 2026, revealing that Agent harnesses significantly impact GPU energy usage through their prompt construction methods. Prompt Caching is identified as the largest source of efficiency, where a cache miss (defined as less than 10% of the prompt being cached) roughly doubles energy costs. For the GLM-5.3 model, cache misses consume approximately 2.5x the energy of fully cached requests, while DeepSeek V4.1 Flash sees a 91% increase in energy consumption.

Impact of Cache Breaks on Latency and Cost

Small changes at the beginning of a prompt, such as timestamps or reasoning effort adjustments, trigger full recomputation. In long sessions exceeding 128k tokens, rare turns that require re-reading more than half the prompt account for 75% of re-read tokens at that depth. These specific turns average 230k tokens of re-reading, generating 600x the Prefill cost of a standard turn (median 370 tokens). Consequently, Time to First Token (TTFT) spikes from the typical 0.8–2.8 seconds to 41 seconds during these large re-reads.

Common Causes of Cache Breaks

Lab tests on clients like OpenCode, Pi, omp, and Kilo Code showed that 88% of requests were pure appends, but the remaining 12% suffered cache breaks due to harness modifications:

• OpenCode Plan-mode reminders: Adding/removing reminders in previous messages caused full re-reads of 330–2,340 tokens. OpenCode 2.x resolved this by separating reminders into new messages.
• omp File Pruning: Replacing previous file reads with [Superseded...] altered the prompt mid-stream, causing 5,846/15,520 tokens to be re-read. This can be mitigated by setting compaction.supersedeReads: false.
• Dynamic Elements: Changes to Reasoning Effort, MCP server connections, Skill additions, or Date changes in the system prompt frequently break caches. OpenCode 2.x mitigates date and tool list issues by appending notes rather than modifying the original prompt structure.

Strategic Recommendations for Harness Developers

To maximize cache hits and minimize energy waste, Neuralwatt recommends:

• Fixed System Prompts: Exclude volatile elements like dates or git status from the system prompt.
• Append-Only Architecture: Maintain byte-identical requests for retries and add state changes as snapshots after the history.
• Stable Tool Lists: Keep tool definitions consistent across modes; use text instructions for mode control instead of changing the tool list.
• Late Compaction: Perform compaction late in the session to reuse existing prefixes and induce cache hits during summary calls.

Neuralwatt is implementing prefix-aware routing to improve hit rates by 0.1–0.2 percentage points and plans to upstream fixes to open-source harnesses. They are also researching server-side techniques to reuse cached work even when clients make minor edits to the prompt's beginning.