Compiling Evaluations from Context

For every LLM call, a Context Manifest is created first. It contains the intent of the call, input types, surrounding code and documentation, constraints, examples, and remaining uncertainties.

Then a separate meta evaluator builds an Execution Contract from this manifest, defining success criteria and compiling them into executable scorers. Rather than leaving both the creation and verification of the evaluation stack to the same agent, this approach separates roles using a prosecutor-auditor-judges pattern.

The core idea is to avoid the limitations of the "just have Claude write the evals" approach.

• It can start even with little initial trace data.
• It first leverages local context in the codebase, such as prompts, policies, tests, and docs.
• Humans don't need to manually design the entire evaluation stack from scratch.
• It explicitly surfaces context gaps such as missing policies, ambiguous prompts, and gaps in test data.

The workflow is simple.

1. A code agent investigates the LLM call site and creates a manifest.
2. A system like Scorable builds a judge from that manifest.
3. If information is missing, it returns a list of gaps.
4. The developer or agent fills in those gaps and runs it again.

The advantages of this approach are privacy, separation of roles, reduced difficulty of initial adoption, and gradual improvement of evaluation quality. On the other hand, it also notes limitations: if the initial manifest is wrong, recovery can be difficult, and cases relying on external observability systems are not fully covered either.