Eval Engineering Skill: Building Evals with Repository Context and Traces

Eval Engineering Skill is a skill that helps coding agents build evals by leveraging repository context and execution traces. It identifies agent structure—prompts, models, tools, hooks, etc.—from the repository, and analyzes actual tool call patterns from traces collected via tools like langsmith-cli.

Instead of simple batch generation, it adopts a user interview approach, receiving feedback on proposed eval directions and iteratively gaining approval. Users decide which tools to actually execute, and whether tools that incur costs or production writes should be simulated instead.

The final output is an executable eval task in Harbor format. Each task consists of the following three components:

• Instruction: The task description delivered to the agent
• Environment: A Dockerfile containing tool installation and data configuration
• Verifier: A verifier that grades whether the task was completed

It was rare for the first version of a verifier to be the final one. By analyzing both the agent trajectory and the verifier trajectory together after running an eval, you can discover reward hacking (such as excessive citation of irrelevant sources, or claiming actions that were not actually performed) and improve the task, environment, and verifier.

It was validated against the chat-langchain documentation Q&A agent, using questions extracted from actual traces along with a verifier based on golden answer strings and cited documents.