Human: Judgment Beyond the Score

Part 3 of the AI-native hiring series covers what happened after Part 2's machine classified over 400 people and generated customized interview guides, when 20+ interviewers directly met over 100 candidates in person over 5 days.

The Functional Gate filtered whether results were correct, but code alone made it hard to distinguish who had actually thought it through themselves versus an output that AI had produced well. A single submission couldn't confirm reproducibility either, and the Depth score also only inferred thinking through traces left in code and documentation.

The machine extracted evidence from candidates' code to build questionnaires. For example, using a code snippet like the dual-lock strategy in EnrollmentService.java as grounds, it asked why the lock was split into two, and whether it would work even in multi-server deployment, attaching expected answer levels and follow-up questions.

Interview strategy also changed depending on the score combination.

• Ace/Craftsman: For candidates with high Base and high Depth, questions pushed on the architecture's limits and boundaries.
• Hustler: For candidates with strong functionality but possibly shallow thinking, the reasoning behind design choices, consistency between documentation and code, and the basis for decisions were verified.
• Thinker: For candidates weak in build or tests but with high thinking ability, causes of failure, recovery ability, and depth of understanding were checked.

Actual interviews revealed patterns different from expectations. Some people with high functionality scores actually showed deep understanding, while some with high quality scores had submitted code that AI had made well but that they understood only shallowly. Conversely, among candidates who scored low, there were cases showing experienced-level thinking, pointing out limits of concurrency control and even fairness issues after sharding.

Evaluators also needed calibration. Scorecards were managed via Git, and since each interviewer had different leniency bias, each was assigned 4-5+ cases, and scores were calibrated by comparing the tone of scores against comments. Interviews were split into 30 minutes of technical and 30 minutes of human evaluation, and for entry-level candidates, the focus was on learning and adaptation attitude, how they collaborated with AI, and critical acceptance and self-judgment.

The conclusion is clear. Code is only a shadow of intent, so why a particular design was made is revealed in design documents and human conversation. As in what's recently called Context Engineering, explicit context and design documents produce more consistent results than volatile prompts. The more AI takes over tasks, the more humans get to focus on problems that need judgment rather than tools, and hiring too should shift toward reading context and intent rather than scores.