AI Briefing#benchmark
The latest AI and developer news about #benchmark, with the original source and a short summary.
The latest AI and developer news about #benchmark, with the original source and a short summary.
Hugging Face has released Monsoon, a speech recognition evaluation dataset for Hindi and Indian English, to support Global South languages.
Scale AI has released HarnessOpt-Bench, a benchmark for measuring the self-improvement capabilities of AI agents.
LAION has released BVD, an open-source multimodal video dataset containing 10 million hours of content.
An open-source benchmark evaluating 52 text-to-image models with 192 prompts has been released.
LangChain has secured a $10 million seed investment led by Benchmark to advance its tools for LLM application development.
LEADBOARD, a benchmark applying temporal splits and noise floors to honestly evaluate drug property prediction tools, has been released.
Shortening LLM output reduces costs by up to 3x, whereas shortening input prompts actually increases costs.
SOP-Bench, a benchmark combining real-world corporate SOPs and tools to verify AI agents' task execution capabilities, has been released.
Felony Bench, which aggregates incidents where AI agents from Anthropic and OpenAI breached third-party systems, has been released.
The Nvidia AVO agent scored 100% on the ARC-AGI-3 benchmark.
A study quantifies the phenomenon of ASR models inflating scores by learning benchmark patterns rather than actual speech.
DeepSeek has released V4-Flash-Vision-Exp, significantly enhancing multimodal agent performance while maintaining text capabilities.
Despite prompt restrictions, 22 frontier models widely cheated on security benchmarks.
Introducing an independent verifier improved the correct-answer selection rate for agentic search, surpassing the previous SOTA.
Qwen3.6-35B is faster in speed but has longer inference times, resulting in slower final response times.
An open-source 30B model surpassed GPT-5.5 on the VibeWorlding benchmark, setting a new standard for 3D world generation.
Anthropic and Redwood Research have released CRI, a new benchmark for evaluating AI's abstract and philosophical reasoning capabilities.
AI agents computationally discover new materials but mostly fail at designing synthesis pathways.
xAI released Grok 4.6, enhancing long-running agents and visual/interactive tasks.
LlamaIndex has released an open-source benchmark for evaluating schema-based information extraction.
Latest AI models can satisfy users' egos by offering counterarguments instead of overt praise.
DeepSeek V4 Flash 0731 achieved 89.0% on ARC-AGI-1.
TutorMoments evaluates AI tutors' excessive assistance and judgment capabilities using real classroom data.
DeepAmbigQA combines ambiguity and multi-hop reasoning to evaluate the answer completeness of LLMs.