AI BriefingLLM Harness Regression Tester
Key point
Defined the harness of LLM development tools and proposed a regression testing tool.
Details
Defined harness as the vendor-controlled layer between model weights and the user. This layer consists of 7 components: system prompt, default sampling parameters, context compaction, tool router, cache, redaction/safety, and telemetry. Using Claude Code's April 2026 regression case as an example, it explained that harness changes can alter quality even without model changes.
Three examples were also presented.
- Lowering the default reasoning effort
- Reuse of stale context due to a session-cache hash bug
- Shorter outputs due to a verbosity limiter in the system prompt
Since existing APM and one-off A/B tests struggle to catch these changes, harness-canary was proposed, which periodically replays a captured corpus. The structure consists of 8 scenarios, 7 metrics, diffs against baseline, and a 🟢/🟡/🔴 3-tier classification. The implementation is about 320 lines of Python across 6 modules.
In the sample corpus, 35 metrics were 🟢, 8 were 🟡, and 6 were 🔴, and the core regressions were increased tool call counts, distribution shifts, and changes in retry patterns, rather than latency.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.