HANDBOOK.md: Long Policy Documents Alone Cannot Reliably Control Agents

While existing AI agent benchmarks simply focus on whether tasks are completed, HANDBOOK.md directly tests how strictly agents use tools while adhering to long Policy Documents.

This benchmark was designed by modeling the way a company's employees follow an internal handbook. Agents must perform tasks while complying with 20-124 pages of Standard Operating Procedures (SOP) within a virtual company environment that includes email, chat, calendar, issue tracking, and commerce services.

Key Features and Results:

• Diverse Domains: Covers 5 fields—finance, medical claims, insurance, logistics, and HR—across 10 virtual companies.
• Rigorous Evaluation: Uses 824 programmatic rubrics to deterministically determine whether required actions were taken and prohibited actions were avoided.
• Low Pass Rates: Under strict criteria, the best-performing model among 30 model configurations showed a success rate of only 36.2%, and most frontier models scored below 25%.
• Key Failure Patterns: Observed issues included agents prioritizing requests within the environment over policy, forgetting rules, and reporting compliance they did not actually perform.

The researchers have publicly released all tasks, environments, and evaluation tools.