AI Briefing#ai-benchmark
The latest AI and developer news about #ai-benchmark, with the original source and a short summary.
The latest AI and developer news about #ai-benchmark, with the original source and a short summary.
Developed by Ai2 and partners, XYBench contains 8,115 questions to measure how well models recognize when a user's request is based on a misconception and steer them toward the underlying goal.
The new 200-task MKA benchmark, built with Artificial Analysis, evaluates agents on visually complex work across 10 verticals and 83 occupations.
In 386 games of the social deduction game Blood on the Clocktower, GPT-6 Astra achieved a 92% win rate against Claude Opus 4.6, with OpenAI models showing superior critical evaluation of evidence.
The benchmark tests agents building 17 products from scratch, with top models failing on feature intersections and schema errors.
The new open-source Parsimony benchmark quantifies AI coding efficiency by counting net syntax units added, revealing that Gemini 3.1 Pro Preview adds the least code among top performers.
A new benchmark reveals that GPT-6 Luna, the model powering OpenAI's upcoming Decisions API, exhibits severe overconfidence and accuracy drops on multi-step reasoning tasks.
Surge AI has released DAYJOB: Finance, a benchmark evaluating AI agents on complex, long-horizon financial tasks involving corporate finance, banking, and investment.
EEBench, a benchmark evaluating AI's electronic engineering design capabilities, has been released, with Claude Opus 5 taking first place at 61.6%.
GPT-6 Astra outperformed Claude Fable 5.1 in robot block insertion tasks with a 95% success rate and 2.3x lower cost.
A new AI benchmark called TxBench-AB has been released to evaluate the biomedical research efficiency of LLMs.
Multiverse Computing has released Quasar 438B, an inference model that achieved the highest score among European models with 43 points on the Artificial Analysis Index.
Ornith-1.5 has been released as a series of open models ranging from 397B to 9B parameters.
dig.bench is a benchmark of 70 text-based games measuring AI agents' scientific discovery capabilities.
About 12% of questions in major AI benchmarks such as GPQA and MMLU-Pro were found to contain errors, leading to the release of cleaned datasets.
Vercel has released DeepsecBench, a benchmark that evaluates AI models' ability to discover cybersecurity vulnerabilities.
Legora BAR evaluates AI models' agentic reasoning capabilities using real legal cases rather than synthetic data.
Meta's next-generation model Watermelon is in development and has recorded benchmark performance on par with GPT-5.5.
GeneBench-Pro, a new benchmark for testing AI performance in genetics, biology, and scientific research using complex real-world datasets, has been released.
DeepSWE, a new code benchmark designed to prevent data contamination and reflect real-world software engineering complexity, has been released.
OpenAI has released LifeSciBench, an expert-validated benchmark designed to reflect real-world life sciences research processes.
China's Weibo has released a 3B-scale VibeThinker model that has become controversial after posting benchmark scores that surpass ultra-large models.
Kaggle has launched a feature that lets users create and run AI benchmark tasks directly from their local development environment.
Beyond raw performance, 'intelligence per dollar'—a measure of AI model cost efficiency—is emerging as a new industry standard.
In CAISI's evaluation, DeepSeek V4 Pro lagged about 8 months behind the leading US model.