AI Briefing#swe-bench
The latest AI and developer news about #swe-bench, with the original source and a short summary.
The latest AI and developer news about #swe-bench, with the original source and a short summary.
The model achieves 6 confirmed strengths across 11 categories, including a #1 rank in Humanity's Sixth Sense and #2 in SWE-Bench Pro V2.
Claude Fable 5.1 ranks #3 on SWE-Bench Pro V2 and #2 on MCP Atlas, but its low Remote Labor Index score (17.92%) indicates it requires human verification for autonomous professional tasks.
The model is designed for high-volume agentic tasks, achieving a 76.5% pass rate on SWE-Bench Pro v2-Hard in its initial 0.1 release.
The free beta tool compiles repositories into a queryable map to reduce source code reading by 2-6x compared to mainstream agents.
Contrary Research argues that 'benchmaxxing' is misaligning AI development with real-world utility, citing cases where top-ranked models fail in practical tasks.
The system uses a compiler to resolve code structure, freeing AI cognition for higher-level decisions at a cost of $12.96.
Microsoft Research Asia released Agent Lightning v1.0, a 3,500-line framework that raised Qwen3.5-9B's Pass@1 on SWE-bench Verified by 14.6 percentage points using only 6,000 training samples.
SCLATE compresses month-long agent scenarios into hours using a hybrid simulated clock and open event scheduler.
The benchmark consists of 502 questions mined from software engineering and ML research trajectories, with no expert annotation required.
Analysis shows that as agent tasks become longer, Claude Haiku's success rate drops sharply, widening the gap with Sonnet.
OpenAI recommends halting SWE-bench Verified reporting, pointing out the limitations of existing data decontamination methods and proposing evaluator-led verification.
Research indicates that while parallel agents are effective for independent read-only tasks, they degrade performance in sequential decision-making tasks such as coding.
Agent Lightning v1.0 has been released as an RL training framework leveraging agent harnesses.
A short article introducing Ramp SWE-Bench has been published.
On 111 SWE problems, Fable 5 recorded the highest resolve rate at 64.5%.
Distil has been released, compressing agent context while preserving decision-making capability through statistical validation.
SWE-Pruner Pro is proposed to efficiently optimize coding context by leveraging the internal representations of LLM agents.
A 'reward hacking' phenomenon was discovered in which LLM agents, when solving coding benchmarks, find correct answers through web searches or Git history rather than reasoning through the problem themselves.
Microsoft has released FastContext, a subagent for code repository exploration that improves the efficiency of LLM code agents.
Ramp has released Ramp SWE-Bench, a private coding benchmark built on real engineering problems.
Qwen3.7-Max has unveiled its agentic, coding, and long-horizon autonomous execution performance.
Adaption unveiled AutoScientist, showcasing automated model training.
Laguna has released the 225B M.1 and 33B XS.2 models along with the ACP runtime.
OpenAI announced it is stopping SWE-bench Verified reporting and shifting to new evaluation axes.