AI BriefingBuilding a Harness for AI Agent Quality Verification
Key point
This case study analyzes the construction of an automated harness that verifies a 55,000-line FPS game generated from a single prompt.
Details
Claude of Duty is a project that built a 55,000-line FPS game based on Three.js using just a single prompt. The core of this project lies not in the game itself, but in the Verification Harness that judges the quality of code written by AI agents using measurable metrics rather than subjective preferences.
Key Verification Tools and Mechanisms:
- imagediff.mjs: A gate that detects regressions at the pixel level.
- profile.mjs: Analyzes frame time distribution (p50, p95, p99) to identify performance bottlenecks.
- baseline.mjs: A reproducible capture tool that guarantees bit-identical results.
Agent Collaboration and Management Rules: ARCHITECTURE.md is used as the sole coordination mechanism to prevent conflicts between agents.
- Directory Ownership: Each agent is restricted to modifying only its designated directory.
- Dependency and Randomness Control: External package usage is prohibited, and the use of deterministic random numbers (
ctx.rng) is enforced to ensure reproducibility. - Module Management: Interactions occur via
ctx.get()at runtime to ensure the safety of parallel tasks.
Key Takeaways: Average frame metrics (p50) can mask performance issues, making distribution analysis such as p99 essential. Furthermore, a Deterministic environment is a prerequisite for meaningful agent evaluation.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.