AI BriefingDialFork Benchmark Released to Measure Procedural Hallucination in LLM Agents
·2026.11.05 19:44
Key point
The benchmark includes over 36k simulated dialogues across 24 tasks and 12 domains to detect when agents drift from required conversational procedures.
Details
- DialFork is the first benchmark designed to measure procedural hallucination, a failure mode where LLM agents stop following their defined conversational workflow despite maintaining factual accuracy and correct tool usage.
- The dataset contains over 36k simulated dialogues spanning 24 tasks and 12 domains, created by forking correct workflows with specific drift-inducing instructions at chosen turns.
- Three evaluation tasks are defined: identifying the specific turn where drift starts (Task 3), measuring procedural distance between a correct dialogue and its hallucinated versions (Task 2), and comparing distributions of correct versus hallucinated dialogue sets (Task 1).
- All resources are open-source, including the dataset, code, human judgments, and a public leaderboard on Hugging Face for tracking progress on the three tasks.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Herdr releases native desktop app for managing terminals and coding agents