Large-Scale Verification for Agent-Based Development

As software engineering shifts toward an async approach, verifying the massive number of PRs (Pull Requests) generated by AI agents like Devin has emerged as a key challenge. Recently, Cognition has been focusing on technology to prove that agent-written code actually works, as asynchronous agent tasks triggered by events or automation surge in volume.

Devin verifies its own work within a cloud virtual machine (VM) using its Computer Use capability. It takes screenshots, moves the mouse, clicks, and types, operating much like an engineer who directly runs and tests an app. This allows multiple instances of Devin to run in parallel to verify complex changes simultaneously, enabling large-scale scaling that would be impossible in a local environment.

To increase the reliability of verification, Devin first writes a Test Plan based on the source code when entering test mode. This prevents errors where the model assumes nonexistent paths or tests the wrong parts, and helps it successfully verify the targeted functionality through accurate configuration even in complex environments where multiple services are interconnected.