Analysis of the 'Home Advantage' Phenomenon in Document Extraction Benchmarks

An analysis of 13 leaderboards related to document data extraction (OCR/parsing) confirmed a 'home advantage' phenomenon where models associated with the benchmark creators or sponsors achieved overwhelmingly high rankings.

Key Statistics (as of September 1, 2024)

• In all 10 benchmarks where manufacturer/sponsor-associated models participated, they ranked within the top 3.
• Among these, they took 1st place in 7 benchmarks.
• Examples: LlamaParse Agentic (84.88 points) ranked 1st in LlamaIndex's ParseBench, and Reducto Deep Extract ranked 1st in LongExtractionBench sponsored by Reducto.

Cause Analysis This phenomenon is attributed to the following structural factors:

1. Problem Definition Bias: Designing benchmarks based on failure types that their own tools handle well.
2. Configuration Asymmetry: Running their own tools with optimized settings while running competitors' tools with default settings.
3. Performance Check Tooling: Using their own benchmark scores as implicit optimization criteria during releases.
4. Publication Filtering: Not publishing benchmarks where performance is low.

Implications and Recommendations There are cases such as HunyuanOCR or olmOCR 2, where models receive high scores on their own benchmarks but see a sharp decline in performance on independent benchmarks. Therefore, when referring to leaderboards, it is recommended to exclude scores for manufacturer products, cross-verify with independent benchmarks, or test directly with your own data.