AI Briefing#evaluation
The latest AI and developer news about #evaluation, with the original source and a short summary.
The latest AI and developer news about #evaluation, with the original source and a short summary.
This article presents an evaluation framework for verifying the real-world performance and safety of LLMs before production deployment.
Anthropic has launched a $5 million research grant program to independently assess the impact of AI on user wellbeing.
LangChain has released a two-stage pipeline that automatically generates environments and tasks for agent evaluation.
LEADBOARD, a benchmark applying temporal splits and noise floors to honestly evaluate drug property prediction tools, has been released.
A study quantifies the phenomenon of ASR models inflating scores by learning benchmark patterns rather than actual speech.
Google has released agents-cli, which injects ADK development, evaluation, and deployment knowledge into coding agents.
LangSmith has redesigned its homepage and enhanced resource management features to optimize developer workflows.
LangSmith has introduced a feature that allows LLM evaluation models to improve themselves by learning from human feedback.
Datacurve's DeepSWE benchmark found that existing coding model leaderboards have significant scoring errors.
To overcome the limitations of existing LLM judges, an agentic approach capable of verifying external systems — the **Agent Judge** — is needed.
LangSmith Engine analyzes failure patterns in AI agents and automates the development cycle from root cause diagnosis to fix suggestions.
Hugging Face has released a leaderboard that compares agent systems' performance and cost together.
LangChain has launched LangChain Labs, an applied research organization focused on continual learning for agents.
A cache key design was proposed that satisfies both reproducibility and variability together in a non-deterministic LLM environment.
Harvey has released LAB, an open-source benchmark for legal agents.
Adding inference-time review to tool-calling agents reduced errors before execution.
AISI assessed GPT-5.5's cyber capabilities as similar to Claude Mythos.
In AISI evaluations, GPT-5.5 showed cyber performance surpassing Mythos Preview.
Qwen has released Qwen-Scope, an SAE-based interpretability tool built on Qwen3 and Qwen3.5.
DeepMind has released ProEval, which reduces GenAI evaluation costs by up to 100x.
OpenAI announced it is stopping SWE-bench Verified reporting and shifting to new evaluation axes.
OpenAI pointed out contamination and flaws in SWE-bench Verified and recommended discontinuing it.
Beyond AI scores and guides, human judgment in interviews was what truly separated real understanding.
It measured the problem of code-editing LLMs excessively rewriting instead of making minimal edits.