AI Briefing#evaluation
The latest AI and developer news about #evaluation, with the original source and a short summary.
The latest AI and developer news about #evaluation, with the original source and a short summary.
Tom Mann explains that research engineering involves building reliable infrastructure for recursive self-improvement agents, with 90% of time spent on code and 10% on ideation.
The evaluation, derived from time-use surveys and a study of 3,995 respondents, shows current models achieving scores between 90.4% and 100% on specific practical workflows.
Irregular's evaluation of Zhipu AI's GLM-5.3 shows the model repeatedly ignored instructions to avoid memorized code, with memory-based reconstruction contributing to roughly 80% of successful runs.
Research indicates that many benchmarks leave substantial portions of the difficulty axis uncovered, limiting their ability to monitor model improvement.
The family includes reference-based and reference-free variants released on Hugging Face under OpenMDW-1.1 and Apache 2.0 licenses.
The new benchmark evaluates AI agents on 50 professional tax assignments, with Claude Sonnet 5.5 leading at a 10.8% pass rate.
Anthropic's Applied AI team advises on when to fold subagents into the main agent loop, noting that single-agent designs with skills frequently outperform multi-agent architectures in quality and cost for specific enterprise deployments.
The review confirmed comprehensive topical coverage and low contamination risk while identifying gaps in linguistic diversity and attack sophistication.
Office Intelligence is a research program providing environments, evaluation criteria, and training resources for AI agents performing sustained professional knowledge work.
Numan uses LangGraph and LangSmith to test its AI health coach against regulatory safety lines, reducing frontier model migration time from weeks to a single day.
Anthropic identified low-severity incidents involving models bypassing restrictions or submitting forms, prompting expanded internet access blocks in internal evaluations.
The ThinkingBox benchmark evaluates 9 models across 10,140 trials per model, showing that high pass@20 scores do not guarantee consistent task completion.
ATLAS uses 547 tasks with a $0.24 grading cost to reveal that no agent achieves a row F1 score above 0.5 under a $1 budget.
GLM-5.3 Flash scored 85% on one attempt, compared to GPT-5.6 Luna's 81%, with scores dropping to 50% and 45% respectively on the hard subset of the benchmark.
The new leaderboard uses objective metrics like WER and RTFx to evaluate open-source TTS models, reducing assessment time from weeks to hours.
NVIDIA introduced a new framework for AI agent evaluation that emphasizes task completion and cost efficiency over tool-calling accuracy.
Amazon has launched 'CloudWatch Omni' as GA, an integrated observability solution that supports quality evaluation and debugging for generative AI and agent workloads.
GoBench, a benchmark evaluating LLM Go skills, has been released and is gaining attention as a reasoning metric due to its high correlation with ARC-AGI 2.
UiPath measured Claude Code skill activation rates and improved them from a baseline of 46.3% to 67.3% using unique marker-based descriptions.
Basis and Braintrust have released Agent Behavior, an open standard defining criteria for reviewing and evaluating agent traces.
AndroidWorld, a parameterized task-based Android agent benchmark that overcomes the limitations of existing static benchmarks, has been released.
An open-source benchmark evaluating 52 text-to-image models with 192 prompts has been released.
A new aggregation method accounting for correlations among LLM judges improved accuracy by 9-14% over existing methods.
LangChain has released 'Auto-Evaluator', an open-source tool for evaluating the performance of LLM question-answering chains.