AI Briefing#swe-bench
The latest AI and developer news about #swe-bench, with the original source and a short summary.
The latest AI and developer news about #swe-bench, with the original source and a short summary.
OpenAI pointed out contamination and flaws in SWE-bench Verified and recommended discontinuing it.
A Princeton/Stanford team has released a 100-line bash-based AI coding agent.
Kimi Vendor Verifier validates implementation deviations across inference providers.
Moonshot AI has released Kimi K2.6 via Chat and API, touting it as open-source state-of-the-art.
On dual RTX 5060 Ti GPUs, Qwen 3.6-35B-A3B hit 21.7 tok/s at 90K context.
Opus 4.7 improved on coding but regressed significantly on long-context reasoning.
The LLM judge favored patches that 'looked like gold,' distorting benchmark scores.
Structural loopholes were revealed in 8 AI agent benchmarks.
8 major AI agent benchmarks are all shown to be vulnerable to score manipulation.
OpenAI pointed out contamination and test flaws in SWE-bench Verified and recommended discontinuing it.
Agent evals measure multi-turn behavior to catch regressions and quality changes before deployment.
Surviving 200,000 evaluations means even agentic benchmarks need multi-tenant infrastructure.
AI21 Maestro boosts SWE-bench performance through structured planning and parallel execution.
The 480B MoE Qwen3-Coder aims for SOTA in coding, browser use, and tool use.
Mistral AI has released Devstral Small 1.1 and Devstral Medium models with significantly enhanced coding agent performance.
With the think tool, Claude improves policy compliance and consistency in complex tool use.
The upgraded Claude 3.5 Sonnet scored 49% on SWE-bench Verified.
OpenAI has released **SWE-bench Verified**, a human-validated benchmark, to more accurately evaluate AI's software engineering capabilities.