AI Briefing#terminal-bench
The latest AI and developer news about #terminal-bench, with the original source and a short summary.
The latest AI and developer news about #terminal-bench, with the original source and a short summary.
The 300B-scale model achieves 46.2% on Terminal-Bench 4.0 using only 64 NVIDIA B300 GPUs over five days.
Factory releases Droid 1.0, featuring Programmatic Tool Calling (PTC) and a 37% reduction in first-request input tokens.
The LangSmith team details strategies for agent evaluation, including using cheaper judges like Jev to cut costs from $28.17 to $0.34 per run.
Replit Agent's new harness design beats the sidekick architecture by 11 points on DeepSWE and 16 on Terminal-Bench, achieving Pareto-efficiency against Astra baselines.
Vercel now supports the Harbor evaluation framework, enabling parallel benchmark execution on Firecracker microVMs.
Cline has released an early version of a desktop app designed to support open-weight models and manage parallel agents.
Artificial Analysis released Intelligence Index v4.3, introducing Terminal-Bench 4.0 and AutomationBench-AA.
LLM-as-a-Verifier v0.2.0 has been released, achieving SOTA performance on agent benchmarks such as Terminal-Bench 2.1 and reducing uncached input tokens by approximately 3.4x through cache optimization.
Terminal Bench 4.0 has been released, with GLM-5.3 showing performance equal to Fable 5 within the margin of error.
While building developer automation tools, I discovered that GPT-5.6 was cheating on benchmarks.
Upstage released Solar Pro 4, an LLM designed to execute document, tool, and artifact tasks to completion.
DeepSeek V4 Flash now runs by default with new weights on AI Gateway.
The LHTB benchmark has been released to measure whether LLM agents can perform complex tasks spanning hundreds of steps in containerized terminal environments.
Anthropic unveiled in London a coding agent that autonomously handles everything from writing PRs to deployment.
Cline has released an open-source agent runtime SDK that powers its entire product lineup.
Laguna has released the 225B M.1 and 33B XS.2 models along with the ACP runtime.
Agent evals measure multi-turn behavior to catch regressions and quality changes before deployment.
How infrastructure resource configuration is set during agentic coding evaluations has a significant impact on the measured results of model performance.