AI Briefing#benchmark
The latest AI and developer news about #benchmark, with the original source and a short summary.
The latest AI and developer news about #benchmark, with the original source and a short summary.
MirrorCode evaluated long-horizon autonomous coding ability and revealed Claude Opus 4.7's results.
APEX-Accounting evaluated whether AI can repeatedly produce accurate results in accounting tasks.
A roundup of 10 AI/ML papers covering real-world agent evaluation and token efficiency.
DeepSeek has released the V4-Flash API in public beta with enhanced agent performance.
A new benchmark called HANDBOOK.md has been released to evaluate the ability of AI agents to perform complex tasks while complying with long policy documents.
An automated AI security leaderboard measuring the security vulnerabilities of frontier models has been released.
A new benchmark called Handbook.md has been released to evaluate whether AI agents perform tasks while complying with lengthy policy documents.
SWE-rebench announced a multilingual software engineering benchmark update that goes beyond Python to include 5 languages such as Go, Java, and Rust.
The ARC-AGI-3 leaderboard, which measures the adaptive intelligence of AI agents, has been released.
Perplexity has released WANDR, an open benchmark that evaluates a research agent's ability to collect data at scale.
A Huawei-led consortium released a technical report demonstrating the training performance of DeepSeek V4 models using Huawei Ascend chips.
On the new ActiveVision benchmark, GPT-5.5 scored 10.6% and Claude Fable 5 scored 3.5%, while human participants achieved 96.1%.
After generating and analyzing 1,008 SVGs, no evidence was found that AI labs are overfitting to the famous 'pelican riding a bicycle' benchmark.
GLM 5.2 officially ranked 3rd on the ProgramBench benchmark, which tests reconstructing source code from binary executables.
InternScience released Agents-A1, a 35B MoE agent model that rivals 1T-class models by scaling agent horizon instead of parameters.
According to a new benchmark, Anthropic's Claude, unlike other models, did not make coercive threats to delete subordinate models.
Kimi K3 is the strongest open model available today, but it still trails the closed-model frontier by several months.
A human-annotated LVSum benchmark has been released to evaluate temporal accuracy in long video summarization.
ASCIITermDraw-Bench, which evaluates VLMs' ability to generate and edit text-based diagrams, has been released.
Anthropic unveiled an LLM reasoning tool called Schema, which achieved 99% performance on the ARC-AGI benchmark.
NVIDIA's Nemotron 3 Embed 8B model achieved No. 1 (78.5%) on the RTEB benchmark, and the company also released a lightweight 1B variant and a Blackwell-optimized version.
LG AI Research proposed the QASA benchmark and a multi-step reasoning methodology for deep reasoning on scientific literature.
LG AI Research has released the QASA methodology and benchmark to address complex reasoning in scientific papers.
This piece examines the LLM reliability evaluation methodologies and research trends in complex task performance and efficiency presented at ACL 2024.