AI BriefingSOB Now Verifies Value Accuracy, Not Just Structure
Key point
SOB has released a new benchmark that measures not only JSON parsing but also value accuracy.
Details
Structured Output Benchmark (SOB) targets the limitation of existing benchmarks that mainly look at JSON schema pass rate, by also verifying the actual output values.
- 7 core metrics: Value Accuracy, JSON Pass Rate, Type Safety, Path Recall, Structure Coverage, Faithfulness, Perfect Response
- Target modalities: text, image, audio
- Evaluation method: compares exact matches against ground truth at the leaf-value level, catching cases like invoice totals or date mappings where the structure is correct but the values are wrong.
In the overall results, open-source models performed well, with GLM 4.7 ranking 2nd, right below GPT-5.4.
On the other hand, most models showed 90%+ JSON schema pass rates, but dropped sharply in value accuracy, revealing the gap between structural validity and actual value accuracy.
Detailed rankings, dataset design, and evaluation criteria have been published on the blog, leaderboard, and paper draft, and the paper is currently pre-arXiv submission.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Perplexity Citation Verification: 34.7% of Sentences with Numbers Have Source Discrepancies