AI Briefing#benchmark
The latest AI and developer news about #benchmark, with the original source and a short summary.
The latest AI and developer news about #benchmark, with the original source and a short summary.
Claude 4.7 uses 1.3–1.45x more tokens, while strict instruction following improved slightly.
Qwen3.6-35B-A3B-UD-Q2_K_XL handled 58 tool calls at a 98.3% success rate on a 16GB VRAM laptop.
A locally run Qwen3.6-35B-A3B drew a better pelican SVG than Opus 4.7.
Claude Opus 4.7 scored 74.0 on the structural biology benchmark, significantly ahead of 4.6.
engram v1.0 claims to have reduced token usage by 88% by intercepting Read calls to compress context.
On an M2 Ultra 64GB, Qwen3.5-35B-A3B Q8_K_XL achieved 63 t/s in generation.
A change in the search strategy of a Klondike Solitaire simulator has pushed the win rate up to 8.590%.
Comparing Vulkan and ROCm on a Strix Halo 128GB machine showed the winner varying by model.
A case of running Qwen 3.6 35B on an RTX 2000 Ada 16GB at 400 t/s prompt and 24 t/s generation.
Claude Opus 4.7 beat Claude Opus 4.6 by 1 keystroke in Vim Golf.
ParseBench is a benchmark that evaluates parser accuracy on your own documents using 5 metrics.
Using turbo3, achieved about 40 tok/s running Qwen3.6 35B A3B at 260k ctx on a 12GB VRAM 3080.
Even the same input can result in actual API costs differing by several times depending on the tokenizer.
A benchmark running MiniMax M2.7 AWQ on 8x MI50 32GB hit a peak of 64 tok/s.
A summary of the performance and limitations of local music generation on Apple Silicon by running ACE-Step 1.5 with MLX.
In MRCR v2 8-needle at 1M tokens, Opus 4.7 scores 32.2%, significantly lower than 4.6's 78.3%.
Opus 4.7 improves text processing performance with an updated tokenizer, but the number of tokens may increase.
Claude Opus 4.7 significantly outperforms Opus 4.6 in coding, long-horizon tasks, and high-resolution vision.
Gemma 4 31B produced better results than competing models with fewer tokens in an image-based 3D geometry test.
llama.cpp's ggml added graph_reused to reduce CUDA's O(n) change checks.
DFlash doesn't speed up as much as expected with MoE, quantization, concurrent use, and long context.
Compared MMLU scores of GGUF quantized models in a 24GB VRAM environment.
Benchmarks illustrate where file-based storage alone is enough and where the line for actually needing a DB gets drawn.