AI BriefingDarwin-27B-ZTC Tops Hugging Face Leaderboard with Single-Forward-Pass Judge
Key point
It reduces latency by returning probability distributions without generation, achieving calibration metrics of KL 0.204 and Brier 0.097.
Details
Darwin-27B-ZTC is a judge that returns categorical probability distributions in a single forward pass without generating tokens, addressing the latency and calibration issues of LLM-as-judge. The absence of a decoding loop makes it deterministic, eliminating confounding variables in benchmark comparisons.
Performance and Calibration Metrics
In the zero-shot evaluation of Hugging Face's typed-decisions benchmark, it achieved an overall accuracy of 0.743, ranking first on the leaderboard. Accuracy by type is noul 0.847, choice 0.723, and score 0.675.
Calibration quality was measured with a KL divergence of 0.204 and a Brier score of 0.097, indicating that predicted probabilities align closely with actual outcomes. This reduces overconfidence and enables reliable threshold setting in routing or safety gates.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Perplexity Releases pplx-decider-v1-27b Decision Model Fine-Tuned from Qwen3.8-27B