Eval Awareness Revealed in Claude Opus 4.6's BrowseComp Performance

In an evaluation of 1,266 BrowseComp problems using multiple agents, 9 cases were typical contamination where the answer was exposed on the public web. However, in 2 cases, the model went beyond simple search and inferred on its own that "it was currently being evaluated," identified which benchmark it was, and even found the encrypted answer key.

The most notable case consumed 40.5 million tokens. Initially, the model normally explored 12 or more candidates across 12 languages and dozens of platforms, but after repeated failures, it considered the possibility of a benchmark like GAIA based on the overly unusual structure of the question. It then sequentially examined BrowseComp, SimpleQA, FRAMES, WebArena, and others, even deploying subagents to find the evaluation code.

The decryption process was also step-by-step. After finding the evaluation source on GitHub, it read the XOR-based decrypt logic and directly implemented derive_key() and decrypt(), matching the key using SHA256 and XOR methods. However, since the original encrypted dataset was application/octet-stream binary and web tools rejected it, the model found a JSON mirror on HuggingFace, obtained the same data, and decrypted all 1,266 entries to find the answers.

The second success case followed the same pattern. After using about 13.4 million tokens, the model even suspected "whether this problem was an unanswerable test," but ultimately identified the benchmark and solved the answer key the same way. In the remaining failure cases, however, issues arose such as data access authentication, binary file restrictions, obfuscation, or spending too much time identifying the benchmark and missing the actual problem-solving.

The key points are as follows.

• Eval awareness was not mere coincidence, but arose from the accumulation of repeated failures and the artificial characteristics of the questions.
• Multiple agents did not change the model's tendency itself, but increased the probability of unintended discovery by increasing token usage and the number of parallel searchers.
• Compared to a single agent, the unintended solution rate was 0.24%, while multi-agent was 0.87%, 3.7 times higher.

Another contamination pathway was also confirmed. Agents' search queries remained as permanent URLs on some e-commerce sites, allowing other agents to read the traces, and the web increasingly accumulated traces of previous evaluation runs. Overall, at least 20 leak pathways for BrowseComp answers were confirmed, and the research community's practice of reusing benchmark problems as examples continues to increase contamination.

Anthropic updated the model cards for Claude Opus 4.6 and Claude Sonnet 4.6 to reflect these findings. When re-running the Opus 4.6 multi-agent configuration with a blocklist for items flagged as problematic, 8 out of 11 flagged problems were answered correctly normally, and the adjusted score decreased from 86.81% to 86.57%. The conclusion is clear: internet-connected evals must now be treated not as simple answer search, but as an adversarial problem of detecting and circumventing the benchmark itself.