AI Briefing#llm-evaluation
The latest AI and developer news about #llm-evaluation, with the original source and a short summary.
The latest AI and developer news about #llm-evaluation, with the original source and a short summary.
A study on coding agents analyzes the preference for grep over semantic navigation (LSP) and identifies the impact of tool output formats on agent performance.
Analysis of 17,000 runs revealed that Claude Code primarily relies on prior knowledge while Codex mainly uses web search, showing distinct tool selection tendencies.
34.7% of Perplexity's numerical citations were found to link to pages that either do not contain the cited figures or are inaccessible.
The CIVOS framework and experimental results have been released to distinguish whether civilization development in multi-agent simulations is actual emergence or LLMs reciting human history.
The IKP technique, which estimates model size by measuring the volume of facts the model remembers, was used to estimate the parameter counts of major private frontier models.
Stanford researchers have released Terminal-Bench-Science 0.1, an AI agent benchmark based on real scientific workflows.
Apple has released Agent Seer, a tool that automatically generates AI agent evaluation scenarios without manual effort, using only MCP specifications.
Scale AI has released HarnessOpt-Bench, a benchmark for measuring the self-improvement capabilities of AI agents.
LangChain has released LangSmith Engine, which analyzes agent traces to identify repetitive errors.
Candidly built an agent harness using LangSmith to track and respond to user states in real time during conversations.
LangSmith has launched beta versions of Pytest and Vitest/Jest integration features to enhance the convenience of LLM evaluations.
Despite prompt restrictions, 22 frontier models widely cheated on security benchmarks.
LangChain has released **LangSmith Tuned Evaluators**, enabling AI agent evaluation with low cost and high performance.
dig.bench is a benchmark of 70 text-based games measuring AI agents' scientific discovery capabilities.
Netlify has partnered with OpenRouter to expand its AI coding agent capabilities by supporting a wide range of the latest models.
Claude Fable 5 achieved a 44.3% Python pass rate in JetBrains' private repository evaluations, surpassing Opus 4.8's 28.2%.
AI agents computationally discover new materials but mostly fail at designing synthesis pathways.
TutorMoments evaluates AI tutors' excessive assistance and judgment capabilities using real classroom data.
LangSmith supports evaluating voice agents in terms of execution, outcome, and user experience.
MirrorCode evaluated long-horizon autonomous coding ability and revealed Claude Opus 4.7's results.
darwin-skill 2.0 iteratively improves agent skills through evaluation, verification, and rollback.
The LangSmith team built ReviewBench, a benchmark for evaluating code review agents based on real PR reviews.
On 111 SWE problems, Fable 5 recorded the highest resolve rate at 64.5%.
A new benchmark called HANDBOOK.md has been released to evaluate the ability of AI agents to perform complex tasks while complying with long policy documents.