AI Briefing#benchmarking
The latest AI and developer news about #benchmarking, with the original source and a short summary.
The latest AI and developer news about #benchmarking, with the original source and a short summary.
Comparing 10 image models on an M1 Max 64GB revealed differences in non-English cultural accuracy.
In Tesla P40 benchmarks, the saw-family KV quantization was the most stable.
The RTX 6000 Pro was significantly faster than the 4080 Super at Qwen3.6 27B inference.
AI eval costs are surging, making evaluation a new compute bottleneck.
On the OnePlus 12's Snapdragon Hexagon NPU, llama.cpp achieved 4.5–12.5 tok/s depending on the model.
Tencent's new model has improved, reportedly with some help from Anthropic.
Native NVFP4 in llama.cpp b8967 boosted prefill speed by up to 68%, but generation speed stayed nearly the same.
In a benchmark of 24 dev prompts, 'be brief.' performed almost identically to caveman.
Measured Qwen 3.6 KV cache performance differences by format from 0 to 1M context on M5 Max.
A hackathon was held where participants use OpenAI Codex to optimize Mac Metal kernels.
Compared the performance of Qwen 3.6 27B's BF16, Q4_K_M, and Q8_0 quantizations.
A llama.cpp benchmark comparing ROCm 6.4.2 and Vulkan on the 6900 XT was published.
On an M5 Pro 64GB, Qwen 3.6 35B A3B was faster and more accurate than the 27B.
In Claude Code, Opus 4.5, 4.6, 4.7, and Sonnet 4.6 were compared by cost and tokens across effort levels.
With llama.cpp ngram-mod speculative decoding, Qwen-3.6-27B speed rose to 136.75 t/s.
Beyond AI scores and guides, human judgment in interviews was what truly separated real understanding.
In a 2 x RTX Pro 6000 environment, Qwen3.6-27B delivered the best performance, while Qwen3.6-35B-A3B showed higher tps.
They released new benchmarks and a harness tailored to the multi-turn, tool-calling patterns of agentic agents.
Kimi Vendor Verifier validates implementation deviations across inference providers.
A case study of pushing an AST interpreter up to 1.9x faster than CPython through 21 stages of optimization.
This is a benchmark result running MiniMax M2.7 4bit on dual Asus GX10 at about 100W.
Kimi has open-sourced a tool for verifying inference accuracy across open-source model deployment environments.
In a 4x RTX 3090 test, MoE Qwen models violated strict bash rules more often than a dense 27B.