Post-Mortem on Three Recent Incidents

From early August to early September, three separate infrastructure bugs overlapped, causing Claude's response quality to intermittently degrade. Anthropic explained that it never lowered model quality due to demand, time of day, or server load, and that the issues all stemmed from infrastructure bugs.

The first issue was a context window routing error. On August 5, some Sonnet 4 requests were mistakenly sent to servers meant for the 1M token context window, with initial impact at around 0.8%. Later, a load balancing change on August 29 caused more short requests to be misrouted, increasing the impact, and at peak impact on August 31, 16% of Sonnet 4 requests were affected. The routing behaved in a sticky manner, so once a request went to the wrong server, subsequent requests were also likely to follow the same path.

The second issue was output corruption. On August 25, a misconfiguration was deployed to Claude API's TPU servers, causing errors during token generation, and in rare cases, Thai or Chinese characters appeared mixed into English prompts, or obvious syntax errors appeared in code. This issue occurred across Opus 4.1, Opus 4, and Sonnet 4, and Anthropic rolled back the change on September 2 and added an unexpected character output detection test to the deployment process.

The third issue was approximate top-k XLA:TPU miscompilation. On August 25, code was deployed to improve the token selection logic, which triggered a latent bug in the XLA:TPU compiler. This bug was confirmed in Haiku 3.5 requests, and it was also believed to have possibly affected some Sonnet 4 and Opus 3 requests. Anthropic switched to exact top-k and standardized some operations to fp32 precision, and is separately working with the XLA:TPU team on a compiler fix.

The core of the problem was not a single cause, but that different bugs overlapped, blending their symptoms together. Anthropic stated that in a structure where multiple hardware platforms—TPUs, GPUs, Trainium—along with Bedrock, Vertex AI, and its own API are operated together, verification is extremely difficult under the premise that platform-specific implementations must produce identical quality. In particular, since models often recover well from isolated mistakes, existing evaluations alone were not enough to sufficiently catch the quality degradation.

Visibility was also a major issue. Due to internal privacy and security constraints, conversations that users did not directly report could not be easily accessed by engineers, delaying reproduction and diagnosis. Anthropic stated that it will strengthen more sensitive evaluations, continuous quality evaluation on actual production systems, and fast debugging tools that do not compromise privacy. It also emphasized that user feedback, the /bug command, thumbs down in the app, and reports to [email protected] remain important.