AI Briefing#benchmark
The latest AI and developer news about #benchmark, with the original source and a short summary.
The latest AI and developer news about #benchmark, with the original source and a short summary.
The 1.7B model scores 59.8 on SWE Verified and 85.3 on tau2-bench, closely trailing the 4B variant's 62.8 and 86.9.
The new public benchmark found that general-purpose models significantly underperform specialized systems in detecting wrinkles and classifying skin types.
The upgraded 560B-parameter model features a 1M-token context window and improved performance on reasoning and automation benchmarks.
With LLM weights frozen, it achieved 87.88% on the GAIA validation set by leveraging a case bank.
StepFun released Step Code, derived from Pi, with built-in sub-agent and web deployment capabilities.
The new benchmark evaluates AI agents on 50 professional tax assignments, with Claude Sonnet 5.5 leading at a 10.8% pass rate.
Execution speed increased 4.4x and cost decreased 6.2x between July and October 2026, with top models achieving 45-48% task scores.
The new model offers 50% lower standard API pricing than GPT-5.6 Terra and GPT-6.1 Sol, while scoring 15 percentage points higher on the Agents' Last Exam benchmark at one-third the cost.
The 2.8 trillion parameter model ranks 4th out of 215 models on the index but is noted for slow output speed and high verbosity.
The 33B model serves as a teacher for upcoming smaller agents and outperforms Qwen3.8-27B on Terminal-Bench 2.1 but trails on SWE-Bench Pro.
A training-free method using five coordinated model instances raises open-weight Qwen3.8-27B accuracy from 66.8% to 92.4% on hard coding tasks.
The pipeline reduces average input tokens from 105,183 to 4,886 while maintaining 96.0% accuracy on LongMemEval, outperforming full-history baselines.
Standard instruction-following rewards cause models to collapse to short, empty answers; LIFT-Bench fixes this by scoring content alongside formatting constraints.
The open-source tool unlocks hardware limits on GA102-GA106 chips, boosting BF16 GEMM throughput by up to 81% and LLM prefill speeds by 14-36%.
The study benchmarks KV cache limits and conversation state management strategies for agent model switching.
The method uses executable database state checks as binary rewards to fine-tune agents, reducing the cost per dependable task from $24.36 to $8.00 for Qwen3.8-27B.
The 816B parameter model scores 39, slightly below the 180B Qwen3.8-Flash-Next which achieves a score of 40.
Open-source engine Kapitsa matches Recursive's native performance on short contexts and leads by 1.30x at 128K context on Apple Silicon.
Ant Ling's upcoming 1019 checkpoint raises the Artificial Analysis Intelligence Index score from 41 to 44 and improves AutomationBench-AA performance to 68%.
Cohere releases Rerank 5 Pro and Fast with Context Focus, reducing token usage by 41% on deep-search benchmarks and 35% overall, while achieving 4x lower calibration error.
The 13-million-case benchmark tests 15 models and finds that existing debiasing techniques often fail to fix this identity-based vulnerability without sacrificing accuracy.
By replacing REA's 207 MCP tools with an 8KB skill file, the author's self-benchmark showed a 38% reduction in execution time and a 35% reduction in cost.
The update introduces Exa Lite at $4 per 1,000 requests and Dynamic Highlights that use 95% fewer tokens than full page text.
An open-source decision model trained on 2,676 contrastive data points that flip the correct answer by changing just one fact