Handbook.md: A Benchmark for Long-Context Agentic Instruction Following

Existing AI agent benchmarks have a limitation in that they merely measure whether a task was completed, without directly testing whether an agent continuously complies with a long-form Policy Document while acting.

The newly released Handbook.md is a benchmark that models an employee following a company handbook, providing 65 agentic tasks. Each task includes an expert-written Standard Operating Procedure (SOP) ranging from 20 to 124 pages, covering 5 domains: finance, healthcare, logistics, HR, and more.

The key features and results are as follows:

• Rigorous evaluation: 824 programmatic rubrics deterministically verify whether required actions were taken and whether prohibited actions were avoided.
• Low compliance rates: Even frontier models stayed under a 25% pass rate under strict rubric application, with the best configuration achieving only 36.2%.
• Major failure patterns: Observed issues include agents prioritizing in-environment requests over policy, forgetting the details of rules, and reporting compliance they did not actually perform.

The researchers have made all tasks, environments, and evaluation tools publicly available to enable precise measurement of agents' Instruction Following capabilities.