Designing a technical assessment that resists AI

Anthropic's performance optimization team has been running a take-home test where candidates optimize code on a virtual accelerator to screen for skill. More than 1,000 people took it, and the task actually contributed to hiring several performance engineers.

The initial version started as a 4-hour task, later reduced to 2 hours. The problem was parallel tree traversal, and candidates had to draw out increasingly higher performance by leveraging accelerator characteristics such as scratchpad memory management, VLIW, SIMD, and multicore. It also included showing the execution process via Perfetto traces.

But as model performance rose, the evaluation signal broke down. Claude Opus 4 produced better results than most human candidates within the 4-hour limit, and Claude Opus 4.5 even reached the level of top human candidates. While humans might do better with unlimited time, it became difficult to distinguish humans from models on a time-limited take-home.

As a result, the task was redesigned multiple times. The first revision extended the problem to a point that was hard for Claude Opus 4 to use as a starting point—removing multicore in the process and adding a deeper optimization space. At the same time, the time limit was also cut to 2 hours to ease scheduling burden.

Even after that, Claude Opus 4.5 broke through the new version as well. It first applied general micro-optimizations to clear the passing bar, and when it judged that it was blocked by a memory bandwidth bottleneck, additional hints led it to find a trick that exploited the structure of the problem. It then went on to debugging and tuning, reaching a level matching the best human performance within 2 hours.

The next attempt was a harder optimization problem, but this time Claude already had a lot of experience in areas like data transposition and bank conflicts. In the end, the model even found a better approach that changed the entire computation structure, and this problem also failed to secure sufficient AI-resistance.

Finally, the author, in search of a more unfamiliar problem, recalls Zachtronics-style constraint-based programming puzzles. The goal is to find areas where human reasoning has an advantage over Claude's learned experience, but the constraint remains that it also needs to resemble actual work. In the end, the core question is how to design assessments that can reveal human capability even in the presence of AI.