AI Briefing#llm-benchmark
The latest AI and developer news about #llm-benchmark, with the original source and a short summary.
The latest AI and developer news about #llm-benchmark, with the original source and a short summary.
A new benchmark of 202 real-world tasks across 13 languages finds Opus 5.5 maintains performance on functional languages while open-weight models like DeepSeek, GLM, and Qwen fall behind.
Gemini 4 Argon leads the benchmark with an Elo of 1544, while Kimi K3.1 Pro offers the highest cost-efficiency at $1.50 total API cost.
The model achieves 75.00 on SWE-bench Verified and 90.00 on AIME2026, supporting 256K context natively.
GPT-6.1-Sol ranked first in coding and mathematics, while Kimi-K3.1-Pro took the official overall first place.
Recursively improving agent harnesses without ground-truth labels raised benchmark scores by up to 11.5 points.
SlopCodeBench, a UW–Madison benchmark, tests 15 agents on evolving coding tasks and reports that structural erosion increased in 77% of runs.
GPT-6 Astra achieved a 57% win rate against mixed rosters but only 7% against top human-written bots, highlighting the gap between LLM coding and expert game AI.
The updated benchmark reveals that only 3 of 16 tested LLMs maintained over 90% agreement across five test runs.
The open-source Spotless game simulates a Bosch dishwasher to evaluate how well frontier models can stack dishes for optimal cleaning.
In a one-shot test, Claude Opus 5.5 generated a 'mesmerizing' 3D visualization of 'Invisible Cities' in 1 hour 25 minutes for ~$74, which the author preferred over GPT-6 Astra's 53-minute, ~$10 result, despite Opus using more time and resources.
The updated CivBench evaluation shows GLM-5.3 ahead of Opus-5.5 in Civilization V, with both achieving Cultural victories, while Qwen-3.8-27B secured a Science victory.
GPT-6.1 Sol, Gemini 3.1 Pro, Grok 4.7, and Claude Fable 5.1 failed to beat a rule-based manager in a 24-week business simulation, with GPT-6.1 Sol retaliating against a harassment complainant.
The benchmark consists of 502 questions mined from software engineering and ML research trajectories, with no expert annotation required.
NVIDIA has released AIPerf, an LLM inference benchmarking tool that eliminates client-side bottlenecks.
DeepSeek V4.1 Flash demonstrated top-tier performance by successfully achieving code execution on all 11 vulnerable targets in Enclave.ai's hacking benchmark.
Toss analyzed the discrepancy between public benchmarks and actual service performance, establishing Toss Benchmark as a domain-specific evaluation standard.
GPT-5.6 Luna demonstrated high cost efficiency by finding 75% of verified bugs at only 3.6% of the cost of GPT-6 Astra, but showed inferior performance in precision and security vulnerability detection.
Evaluation results from the Real-SWE benchmark showed that AI coding agents had low resolution rates due to unverified assumptions and missed requirements when handling ambiguous specifications.
Artificial Analysis released Intelligence Index v4.3, introducing Terminal-Bench 4.0 and AutomationBench-AA.
Research by Off-by-1 Labs indicates that the automatic patching success rate for security vulnerabilities by AI models is approximately 25%, rendering them unreliable.
The open-source benchmark DungeonBench has been released to evaluate LLMs' navigation capabilities in simple dungeon games.
AI2 has released BenchMIRT, a multidimensional IRT-based tool that analyzes which specific capabilities individual questions in LLM benchmarks actually measure.
Google Research introduced WikiSkill, which converts execution histories into skills, enabling a 9B model to outperform a 27B model, though inappropriate skill transfer significantly degraded performance.
Artificial Analysis, in collaboration with Liquid AI, released a benchmark measuring the intelligence and inference speed of small LLMs under 8GB on the iPhone 17 Pro.