AI BriefingSOB Verifies JSON Value Accuracy
Key point
SOB unveiled a new benchmark that verifies not just JSON schema pass rate but also value accuracy.
Details
Pointing out the limitation that existing structured output benchmarks only look at JSON Schema pass rate and types, it presents an evaluation that puts the actually more important Value Accuracy front and center.
- It evaluates using 7 metrics: Value Accuracy (primary metric), JSON Pass Rate, Type Safety, Path Recall, Structure Coverage, Faithfulness, Perfect Response.
- Each leaf value is scored by exact match against a verified correct answer, catching real failures such as wrong numbers or ordering errors.
- It compares multimodal structured output performance covering text / image / audio.
As a result, most models recorded JSON schema pass 90%+, but dropped sharply in value accuracy.
In the overall ranking, open-source models also showed strength, with GLM 4.7 presented in 2nd place, right below GPT-5.4.
Detailed analysis was released in the blog, leaderboard, and paper draft.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Perplexity Citation Verification: 34.7% of Sentences with Numbers Have Source Discrepancies