AI Briefing#benchmark
The latest AI and developer news about #benchmark, with the original source and a short summary.
The latest AI and developer news about #benchmark, with the original source and a short summary.
The startup, valued at $285 million, is backed by Benchmark and Index Ventures and already counts Cognition and Modal as early customers.
Human participants achieve 93.1% accuracy on the new HSS benchmark, while the top-performing model, GPT-6-astra, reaches only 53.6%.
The dataset covers 106 SLMs with 150M parameters or fewer, evaluated on a Runpod B200 under Apache 2.0 license.
The model achieves a 20.92% average Word Error Rate on Arabic benchmarks, outperforming the previous best published result by 2.25 percentage points.
In a test of 2,078 documents, PyMuPDF outperformed Docling with an overall score of 53.6 and a median processing time of 0.47 seconds per page.
Scheduled for release in October under the Apache 2.0 license, this sparse MoE architecture emphasizes inference compute efficiency.
Epoch AI has defined InnovationEval, a benchmark requiring AI agents to develop novel post-training algorithms that outperform a GRPO baseline within a constrained scope of algorithmic changes.
A new benchmark has been released that evaluates tasks by running them 20 times and grading based on the final database state.
The new AFP-GIC framework reduces decoder latency by 18.1% and parameter count by 20.5% compared to DC-VIC while supporting five bitrate operating points.
The model scores 44.86 on Decision Index 0.2.1, ranking #1 among 4B models, and processes 1M-token documents in ~3 seconds via API.
GPT-6 Astra scored 55.0% on Ironclad research tasks, a 32% improvement over GPT-5.6 Sol, while reducing estimated time per attempt by 48%.
GLM-5.3 Flash scored 85% on one attempt, compared to GPT-5.6 Luna's 81%, with scores dropping to 50% and 45% respectively on the hard subset of the benchmark.
The new 7B model achieves 84.83% on the Alyah benchmark and significantly outperforms larger general-purpose models in dialect fidelity and cultural understanding.
The open-source tool achieves 89.8% Recall@1 on LongMemEval-S while running on minimal hardware like an Intel Celeron N4020.
No open model has surpassed Jev, but Cygnet recorded the top score on JevBench.
GitHub has released ReviewBench, an open offline benchmark for AI code review agents, constructed from an analysis of 103.9 million GitHub pull requests.
Out of 44 official leaderboards, there were 0 verified scores, and Chinese organizations accounted for 55.2% of submissions.
GPT-6 Astra solves all 30 standard puzzles, while solve rates drop to 20% on 15x15 grids at best effort levels.
Achieved an AA Index score of 24.1 with an MoE architecture that activates only 3B of its total 35B parameters
A new benchmark, the Telegraph Test, demonstrates that LLMs can read compressed 'cablese' records with accuracy equal to or better than plaintext records, offering 25–49% token savings for machine-to-agent communication.
The new benchmark evaluates 507 stateful business workflows by verifying final database states rather than tool call success, revealing that 67% of agent failures occur despite clean termination.
The new MIT-licensed benchmark evaluates models on finding and fixing real-world bugs in large codebases before users encounter them.
A new benchmark shows Jev-style decision models do not consistently beat LLM-as-a-judge or traditional classifiers, with Qwen3.6-35B leading prompt injection accuracy and Jev leading content safety accuracy.
The 26B parameter model achieves an 85.71% overall accuracy across 11 benchmarks, outperforming its base model Qwen3.8-27B and the Jev baseline.