AI Briefing#agent-evaluation
The latest AI and developer news about #agent-evaluation, with the original source and a short summary.
The latest AI and developer news about #agent-evaluation, with the original source and a short summary.
The new 200-task MKA benchmark, built with Artificial Analysis, evaluates agents on visually complex work across 10 verticals and 83 occupations.
The company outlines three principles for using human-in-the-loop workflows to define quality, evaluate context, and create golden datasets for continuous agent improvement.
The Vercel provider runs agents and task environments in separate Sandboxes, allowing users to inspect task scores and recorded actions.
Seahaven provides isolated, stateful environments with SQLite backends and fixture-based resets for evaluating and training AI agents.
The benchmark consists of 502 questions mined from software engineering and ML research trajectories, with no expert annotation required.
Surge AI has released DAYJOB: Finance, a benchmark evaluating AI agents on complex, long-horizon financial tasks involving corporate finance, banking, and investment.
LangChain's validation of TypeSafe AI's Jev confirmed it is up to 200x faster, 400x cheaper, and more consistent than LLM Judges for classification tasks, though the experiment is in its early stages and requires further verification.
Sierra released Hyper-τ-bench, a benchmark evaluating AI models' agent-building capabilities, confirming that performance improves to 82.2% when collaborating with human engineers.
An open format has been released to document and evaluate the process behaviors of long-running AI agents.
LangChain developed the WikiBench benchmark to measure the performance of its code documentation agent, OpenWiki.
LangSmith Engine has improved agent issue detection performance by more than 2x and fix performance by 25%.
When false documents were injected right before final synthesis, agents' adoption rate rose to as high as 85.5%.
LangChain has released a benchmark framework for its open-source agent harness, Deep Agents.
LangChain has released the 'Eval Engineering Skill,' which automatically generates agent evaluations (evals) by analyzing repository structure and execution traces.
Emergence AI unveiled a multi-agent simulation platform for evaluating long-term autonomy.
It summarizes how to design agent evaluation harnesses that go beyond static benchmarks.
Plano has unveiled Signals V2, which scores agent trajectories without any model calls.
Anthropic added dreaming, outcomes, and multi-agent orchestration to Managed Agents.
8 major AI agent benchmarks are all shown to be vulnerable to score manipulation.
GLM-5 and MiniMax M2.7 delivered closed-model-level performance on core agentic tasks.