AI Briefing#benchmark
The latest AI and developer news about #benchmark, with the original source and a short summary.
The latest AI and developer news about #benchmark, with the original source and a short summary.
The early release of the small model on Gemma 4 E4B scores 0.811 on the Werdykt benchmark and requires basal engine 2.8.0.
Gemini 4 Hydrogen leads the proprietary field with 13.5%, while GPT Astra 6 follows at 9.5% on the same challenging benchmark.
VIDRAFT has released a leaderboard that grades operational forecasts daily and a fusion model on Hugging Face.
OpenAI's unreleased internal model scored 77.0% on ExploitGym, nearly double Anthropic's Claude Mythos 5.2 (39.2%) and significantly outperforming GPT-6 Cyber (52.6%).
Priced at $1.50 per million input tokens, the model targets speed and efficiency for daily coding tasks, distinguishing it from the frontier-focused Gemini 4 Carbon and Argon.
Decider v1.3 achieves 91.6% average accuracy on 63 classification tasks, outperforming previous production classifiers by 6.54 percentage points.
The new benchmark uses 330+ error types to weight parsing errors by severity, correcting previous metrics that penalized formatting over factual errors.
New 2-bit to 6-bit quantizations achieve higher accuracy on agentic and coding benchmarks using 256K and 128K context windows.
Anthropic reports that Claude Fable 5.2 generated complete specs, board layouts, and firmware for three devices that Diode Computers built with no circuit fixes.
Anthropic has released Claude Fable 5.2 and Mythos 5.2, lowering prices and improving output speed by up to 30%.
A new benchmark from Einsia shows that using the strongest model as a coordinator does not guarantee better performance for the worker agents it directs.
In a new benchmark requiring simultaneous fixes for two real bugs, DeepSeek V4.1 Flash achieved a 90% success rate at a cost of 6 cents per task, surpassing GPT-6 Luna.
An Anthropic analysis shows that the effectiveness of agent memory varies by model and instructions, highlighting the importance of verification and restructuring.
The benchmark tests 145 tasks across 9 categories, finding that GPT-6-astra leads in interactive worlds while local mechanisms outperform stateful workflows.
The benchmark evaluates ten models on 273 tasks derived from actual user interactions.
The new models offer up to 30% faster output speeds, improved coding and reasoning capabilities, and reduced false positives in cybersecurity and biology safeguards.
Anthropic analyzed 200,000 Claude sessions and stated that AI significantly accelerates the data analysis and writing stages of scientific research.
The benchmark separates skill learning from familiar setup success, revealing that current systems fail when block positions or stack heights change.
A new benchmark evaluates Codex and Claude on git operations, finding a 359/360 pass rate across the full batch, with GitButler showing the lowest latency and command count in the primary comparison.
Inkling 2 Small achieved the final score of the first generation in just 75 steps of RL fine-tuning.
The benchmark uses 5,800 labeled combinations where every test peptide is at least four edits away from any publicly reported TCR specificity.
The 1.3B parameter model uses a recursive self-improvement loop and pluggable layout analysis to convert PDFs into structured data.
Analysis of 12 months of merged PRs shows the average repo's AI-signed share rose from 2.3% to 10.3%, though the median remains at 0.9%.
Thinking Machines AI introduces Inkling 2 Small, a 594B MoE model with 17B active parameters, initially available with closed weights on Tinker, with full Apache 2.0 open weights planned for next week.