AI Briefing#llm-benchmarks
The latest AI and developer news about #llm-benchmarks, with the original source and a short summary.
The latest AI and developer news about #llm-benchmarks, with the original source and a short summary.
The 2.8T parameter Mixture-of-Experts model features a 1M token context window and achieves 65.7 on Terminal-Bench 4.0.
The open-weights MoE model achieves a score of 45, significantly above the median of 19 for similar-sized models, while offering 230.4 tokens/sec output speed.
A cluster of four M5 Ultra Mac Studios achieves over 36 tokens per second on the 816B-parameter Muse Spark Open model using new RDMA-based distributed inference.
The model scores 59.3 on the internal PI Knowledge Work benchmark at $0.13 per task, undercutting Claude Fable 5 and GPT-5.6 Sol while maintaining competitive quality.
The 3.66B parameter model scores 55.4 on a logic composite, matching Qwen3-8B at less than half the size.
The new open-weight model reduces flagged censorship responses from 89.8% to 2.8% while maintaining general benchmarks within ~1 point of the original.
Offered at $2 per million input tokens and $10 per million output tokens, it is specialized for cybersecurity defense.
GPT-6.1 Sol is available via API and ChatGPT Work/Codex, offering cached input pricing at $0.10 per million tokens and matching GPT-6 Astra on agentic coding tasks at significantly lower cost.
UC Berkeley researchers identified the 'Harness Tax' phenomenon, where achieving the same quality with coding agent harnesses can cost up to 5 times more depending on the harness design.
Artificial Analysis released Capability Indices v1.1, with Claude Fable 5.1 taking first place in all six occupational indices and Grok 4.5 demonstrating high cost efficiency.