One-Third of AI Agent Code Changes Are Not Verified by Tests

A study has been published quantifying the fact that even when AI coding agents run and pass tests, the actual modified code is often not verified. A researcher analyzed 14 real task sessions performed by AI agents in a large open-source monorepo.

The analysis method involved identifying the spec files actually executed from the test command output, calculating the transitive import closure of those specs, and cross-referencing them with the modified source files. Out of a total of 35 modified files, excluding 5 type declaration files that were not execution targets, the results for the remaining 30 files are as follows:

• 47% (14 files): Actually executed by tests
• 33% (10 files): Not executed
• 20% (6 files): Indeterminate (tests not run, dynamic module loading, truncated output, etc.)

In particular, files classified as "not executed" were not connected to the modified code paths, even though tests existed and passed. This signifies a "false positive" situation where changes are not verified despite tests passing. For example, in one session, an agent modified complete-cart.ts and all 5 tests passed, but the spec did not import complete-cart.ts, so the changes were never executed.

The analysis showed that half of the connections between executed tests and modified files (7 files) had a simple structure where the spec file directly imported the modified file. Therefore, the key to verification is not simply asking the agent "Did you run the tests?", but asking "Which of the tests you ran execute this specific file?" and directly checking the import graph.