A Joint Playbook for Trustworthy Third-Party Evaluations

As frontier models evolve beyond simple chatbots into agents that use tools and carry out complex workflows, model performance has come to depend heavily not just on the model itself but on the harness — the environment in which tasks are performed.

For effective third-party evaluation, the evaluation setup must clarify what Claim it is trying to verify and provide evidence that can substantiate the validity of the results. The main targets of verification are as follows.

• Capability elicitation: Can the model actually exhibit the capability in question?
• Safeguard performance: How robust are the safeguards against attacks?
• Comparison: How does performance differ across models under identical conditions?

In addition, factors that could undermine the validity of evaluation results must be thoroughly checked. The main items to check are as follows.

• Reward hacking: Exploiting evaluation metrics to gain scores without actual capability
• Refusals: Cases where the model refuses to answer in order to hide the behavior being tested
• Contamination: Distorted performance caused by evaluation problems being included in the training data
• Broken problems: Degraded performance due to faulty problem setups or tool defects
• Sandbagging: Deliberately lowering performance because the model recognizes it is being evaluated

The choice of harness is particularly critical for systems performing long trajectories. An appropriate harness enables the model to use tools, recover from mistakes, and complete multi-step tasks, allowing the model's actual capabilities to be measured accurately.