AI Briefing#benchmarking
The latest AI and developer news about #benchmarking, with the original source and a short summary.
The latest AI and developer news about #benchmarking, with the original source and a short summary.
ReactBench v1 comprehensively evaluates coding agents' React-writing ability using the React Doctor validator, which has over 400 rules.
The AIDE² system surpassed a model hand-tuned over 2 years through 8 days of recursive self-improvement.
Perplexity has released WANDR, a new open benchmark that evaluates the ability to perform broad data collection and deep evidence verification.
Current alignment benchmarks risk overestimating safety, since models may recognize the evaluation environment or conceal risky behavior.
OpenAI has released GeneBench-Pro, which evaluates AI agents' ability to conduct biological research.
ScarfBench has been released to evaluate whether AI agents can successfully migrate real enterprise Java applications.
OpenAI has released case studies of Genebench-Pro, a benchmark that evaluates the ability to analyze biological data.
A large-scale mutual grading of 55 LLMs confirmed significant evaluation bias according to model family.
An analysis shows that the pattern of the performance gap between Open Weights models and Closed Source models differs depending on the benchmark metric used.
This analyzes the performance and efficiency the GitHub Copilot agentic harness shows across various LLM models and benchmarks.
Hugging Face and Treble Technologies have launched the FFASR leaderboard, which reflects real-life acoustic environments.
It introduces a new benchmarking methodology that measures how much software APIs and documentation affect AI agents' task efficiency.
Recursive unveiled early results from an automated AI research system that achieved SOTA in model training and GPU kernel optimization.
Hugging Face's Papers With Code has been updated to include performance metrics for closed-source models.
A new benchmark and dataset have been released to evaluate speech recognition performance on code-switching speech that mixes multiple languages.
49 mathematicians built a dataset containing research-level math problems and their correct answers.
The METR graph, a key piece of evidence in discussions of AI's rapid progress, has come under criticism for containing serious methodological errors.
CANTANTE improves multi-agent prompt optimization through credit assignment.
Perceptron has unveiled Mk1, introducing a video analysis AI that is 80-90% cheaper than major competing models.
llama-eval, a feature for measuring model performance in local environments, has been added to llama.cpp.
Google is preparing the next upgrade to Gemini Flash around I/O 2026.
Artificial Analysis ranked the anonymous model Peanut #8 on its Text-to-Image leaderboard.
The author puts the probability at over 60% that AI will automate AI research on its own by the end of 2028.
On MBP M3 Max benchmarks, the gap between three Qwen3.6 variants and gpt-oss:120b was revealed.