LlamaIndex Introduces ParseBench-2 with Severity-Aware Metrics for Document Parsing
·2026.11.19 09:00
Key point
The new benchmark uses explicit severity policies to ensure consequential errors like wrong financial totals receive heavy penalties, unlike previous similarity-based metrics that often favored structurally different but factually correct outputs.
Details
- ParseBench-2 introduces explicit severity policies to align deterministic metrics with human preferences, ensuring that critical errors (like wrong monetary values) incur substantial penalties while equivalent representations do not.
- The benchmark expands to 4,317 page-level cases across six dimensions, including new targeted datasets for forms, difficult tables, and formatting, with complete reference tables for all 1,025 charts.
- A Metric Lab prototype allows users to customize dimension weights, severity weights, and error classifications to tailor the evaluation to specific workloads, such as prioritizing table accuracy for financial analysis.
- The system uses a hillclimbing process with coding agents to refine comparison rules against 330 perturbation types, maintaining deterministic scoring while addressing limitations in LLM-based judges that can produce inconsistent preferences.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.