AI Briefing#benchmark
The latest AI and developer news about #benchmark, with the original source and a short summary.
The latest AI and developer news about #benchmark, with the original source and a short summary.
Terminal-Bench scores rose from 31.2 to 42.3, with improved performance on agent and vision benchmarks.
While performance is similar under default settings, the optimal engine varies depending on output length and prefix reuse.
The benchmark evaluates visual agents' ability to construct 3D scenes in Blender using real-world references.
While hpc8a can replace hpc6a, the hpc7g 10-node configuration proved superior in both total cost and execution time.
GPT-6 Astra achieved the highest strict pass rate of 16.3% on SlopCodeBench, while higher effort settings like XHigh often increased costs without improving or even degrading strict pass rates.
The new Sabi-Noema models reduce token generation by 50% on TB2.1 while outperforming GPT 6 Luna-medium on DeepSWE benchmarks.
The new leaderboard evaluates how well embedded signatures survive fine-tuning, distillation, and attacks while preserving model capability.
The new benchmark shows the top model, GPT-6 Astra, scores 65.5 on design creativity compared to a human reference of 92.3.
The new model offers introductory pricing of $2 per million input tokens and $10 per million output tokens, with a 1 million token context limit.
GLM-Legal achieves a 19.2% all-pass rate on Harvey LAB and parity with closed-source models on LegalBench and PLawBench while running on a leaner parameter footprint.
StudentSim outperforms baselines like GPT-5.4 and Maia2 in chess, writing, and math simulations, achieving 90.5% tutor accuracy in reinforcement learning tests.
The benchmark tests nine models across three access levels, finding that the performance gap between open and closed models largely depends on their ability to diagnose issues using only config files and code.
Offers a lightweight decision model specialized for classification and routing tasks at $0.042 per 1M input tokens
The new Apache-2.0 model family, ranging from 0.6B to 27B parameters, achieves state-of-the-art scores on both text and vision boards of the Jev Decision Index.
Evaluation of 17 LLMs revealed hallucination rates of 33–60%, with the absence of a verification layer identified as the primary cause.
A hybrid system combining GPT6-Astra planning with a Jev agent swarm achieved 96% success on standard cases, significantly outperforming the lone model's 57%.
The Apache-2.0 licensed model achieves a composite score of 72.5 on JevBench, surpassing Jev 1.13.
The open-weight Aplomb 1 model achieves a KL divergence of 0.123 and a Brier score of 0.065 on the Typed Decisions leaderboard.
The 5B-parameter model achieves state-of-the-art results on OmniDocBench v1.6 with a score of 96.96 and supports seven distinct parsing families via a unified JSON schema.
Achieves 0.743 accuracy by performing decisions with a single forward pass of a 27B model
The ThinkingBox benchmark evaluates 9 models across 10,140 trials per model, showing that high pass@20 scores do not guarantee consistent task completion.
It reduces latency by returning probability distributions without generation, achieving calibration metrics of KL 0.204 and Brier 0.097.
The 0.8B Jeff model determines whether to think on each turn, reducing Qwen3.8-27B's task time by 32%.
ATLAS uses 547 tasks with a $0.24 grading cost to reveal that no agent achieves a row F1 score above 0.5 under a $1 budget.