AI Briefing#benchmarking
The latest AI and developer news about #benchmarking, with the original source and a short summary.
The latest AI and developer news about #benchmarking, with the original source and a short summary.
Trained Apple's 3B model on both T4 and Mac using QLoRA and obtained equivalent adapters.
On Gemma 4 26B-A4B GGUF, Unsloth quants top the charts by KLD in most cases.
By harnessing an AI agent, a system was built and refined to automatically grade submissions from over 400 people.
METR's time horizon keeps growing, but we examine whether the hourly cost of AI agents is rising along with it.
PrismML has released Ternary Bonsai, a family of 8B, 4B, and 1.7B LLMs using 1.58-bit ternary weights.
Unlike general-purpose benchmarks, Opus 4.7 scored lower than 4.6 in a real-world workflow.
Anthropic has publicly released Claude Opus 4.7, which surpasses Opus 4.6.
pi-autoresearch extends Karpathy-style autonomous experimentation into a general-purpose optimization tool.
On an RTX 5080·64GB Windows 11 setup, Llama.cpp was about 2x faster than LM Studio.
Agent memory reveals bottlenecks across search, cost, and memory management alike.
In 50-turn conversations, the system prompt gets diluted and agents miss safety warnings.
Alongside the launch of Nemotron 3 Nano, NVIDIA unveiled an open evaluation standard built on NeMo Evaluator for transparent verification.
BigCodeArena, a human-in-the-loop platform that evaluates the quality of code generation models based on actual execution results, has launched.
OpenAI has unveiled a new evaluation metric, **GDPval**, which measures the economic value of AI through practical tasks across 44 occupations.
ScreenSuite, an integrated benchmark suite for evaluating GUI agent performance from multiple angles, has been launched.
RAG evaluation should look at real-world document relationships rather than just scores.
OpenAI has released PaperBench, a benchmark that evaluates AI agents' ability to reproduce state-of-the-art AI research.
The Open Arabic LLM Leaderboard 2 has been released to transparently evaluate the performance of Arabic LLMs.
It presents a direction for multilingual and multicultural benchmarking to evaluate LLM performance more fairly and inclusively.
Judge Arena, a benchmarking platform that compares and ranks LLMs' evaluation capabilities, has launched.
It introduces a benchmarking tool that analyzes TGI's performance metrics and how to use it to optimize LLM deployment.
The Open Arabic LLM Leaderboard has been released to professionally evaluate the performance of Arabic LLMs.
A practical guide to minimizing STW by using GOMEMLIMIT instead of GOGC, and reducing GC load through Heap Profile analysis and code optimization tips.
A new open leaderboard that evaluates LLM performance by reflecting the linguistic characteristics of Hebrew has been released.