The Machine: How AI Evaluates Code That Uses AI

Manually reading and grading over 400 submissions was practically impossible. So a terminal-based AI agent was set up as a general-purpose executor, and an evaluation machine was designed to automatically run the entire process—from cloning, security scanning, Docker builds, functional testing, score calculation, to Slack notifications.

The key wasn't building the agent itself, but building the harness for the agent to follow. Instructions were written in Markdown, and output was enforced via JSON Schema to ensure reproducibility. An independent sub-agent was spun up per candidate to prevent context contamination, ensuring each candidate was evaluated in a clean grading room.

The evaluation pipeline consisted of 7 stages.

• Init Stage: Clone the GitHub repo and pin it to the last commit before the deadline
• Security Gate: Check for network requests, OS commands, hardcoded secrets, etc.
• Preflight Gate: Verify validity of README, source directory, and build configuration
• Quality Gate: AI reads the code, documentation, and prompts to score depth across 8 areas
• Functional Gate: Spin up the server in Docker and run test cases
• Scoring Stage: Combine functionality and depth to derive the final score using a 3-Tier model
• Report Stage: Update dashboard, send Slack notifications, clean up Docker

Initially it was Pass/Fail, then S/A/B/C/D/F grades, but without separating functionality from depth of thinking, there wasn't enough differentiation. Eventually the system was organized into Base 100 points + Depth 120 points, with 7 ranks: Ace, Craftsman, Hustler, Thinker, Contender, Rookie, Incomplete. In particular, Hustler and Thinker were ranks created to distinguish cases with strong functionality but shallow thinking, versus cases with stumbling execution but deep thinking.

In practice, rubric tuning was key. Initially there was significant score variance, but adding an evidence checklist reduced the fluctuation. Through 17 grading model changes and 26 resets, the rubric was continuously revised, creating a three-layer feedback loop where a separate agent, after automated evaluation, identified overscored/underscored patterns to improve the rubric itself.

Infrastructure also mattered. n8n, PostgreSQL, Google Sheets, and the GitHub API were connected to handle everything from generating the candidate list, GitHub invitations, collecting submissions, tracking evaluation status, syncing scores, generating the pass list, to inviting interviewers—with a dashboard tracking progress. As a result, the proportion of commits actually running the system far exceeded the commits building the system, and AI grading became not just a simple feature but part of the operational system underpinning the entire hiring process.