AI Briefing#benchmark
The latest AI and developer news about #benchmark, with the original source and a short summary.
The latest AI and developer news about #benchmark, with the original source and a short summary.
The Real-SWE benchmark, which evaluates AI coding agents using proprietary real-world enterprise codebases, has been released, with Fable 5.1 ranking first with a 38.8% resolution rate.
ARC Prize released the ARC-AGI-4 benchmark to evaluate autonomous open-ended innovation and reaffirmed its open-source principles.
ModelRift compared CadQuery and OpenSCAD using agents based on Claude Opus 5, identifying the risk of OpenSCAD's silent failure and CadQuery's explicit failure characteristics, ultimately deciding to retain OpenSCAD.
Meta has released 'WearableQA', a benchmark of 4,084 health reasoning problems utilizing real wearable time-series data from 200 individuals.
Apple researchers have released CapQuiz, a reference-free benchmark for evaluating the information fidelity of video captions.
Research on the DF26 benchmark shows that both humans and existing detection technologies perform at random levels in detecting the latest AI-generated videos.
Q2D-Web has released a large-scale web search retriever evaluation benchmark featuring 190 million documents and 10 languages.
NVIDIA released a NemoClaw-based agent that separates knowledge and judgment, improving benchmark accuracy to 90.9%.
Sierra released Hyper-τ-bench, a benchmark evaluating AI models' agent-building capabilities, confirming that performance improves to 82.2% when collaborating with human engineers.
OpenBMB has released MiniCPM5-2B, which achieved the highest intelligence index score among open-weight models with 4B parameters or fewer.
vLLM released detailed benchmark results comparing the throughput and optimal settings of five speculative decoding methods, including Native MTP, EAGLE-3, and DFlash, on AMD MI300X/MI355X GPUs.
A new mobile agent benchmark named 'MobileWorld' has been released to evaluate capabilities in user interaction and MCP tool utilization.
PyTorchKR summarized 10 recent AI/ML papers centered on three major trends: computational efficiency, autonomous agents, and multimodal reasoning.
The 99.9% score for GPT-6 Astra, presented by OpenAI as evidence of AGI, was achieved using a custom harness, while it scored only 62.7% under the standard harness, leading ARC Prize to reject the AGI declaration.
APL's AI-Eval profile returns 'incomparable' when no applicable bridge exists for scores with different evaluation conditions, without guaranteeing score accuracy or procedural compliance in actual execution.
The 'Auto-BenchMax' data synthesis pipeline, tailored to benchmark scoring methods, has been released and improved the MCP-Atlas score by more than double.
OpenAI has released GPT-6 Astra, significantly improving performance and alignment safety, including achieving 99.9% on ARC-AGI-3.
The Qwen team, in collaboration with Taobao, released E-Commerce Bench, a long-term operational agent benchmark based on one year of real market data.
An evaluation of six open-source AI detection models revealed that four failed to meet the 0.5% false positive rate threshold, with detection rates for humanized text peaking at 42%.
AndroidWorld, a parameterized task-based Android agent benchmark that overcomes the limitations of existing static benchmarks, has been released.
A small transformer trained for 1.5 hours achieved 44% performance on the ARC-AGI-1 benchmark at a cost of 67 cents.
Google Cloud AI Research and others proposed 'EnvHarness', a wrapper layer that dynamically reconstructs static benchmark environments to target agent weaknesses.
Terminal Bench 4.0 has been released, with GLM-5.3 showing performance equal to Fable 5 within the margin of error.
Stanford researchers have released Terminal-Bench-Science 0.1, an AI agent benchmark based on real scientific workflows.