AI Briefingllama.cpp Optimizes Metal for Qwen3.8
Key point
A new llama.cpp branch featuring Metal and n-gram SSD offload optimizations has been released, improving Qwen-3.8-Flash-Next inference performance on Apple Silicon.
Details
An unofficial branch of llama.cpp (metal-qwen4exp-split-ngram) announced that it has optimized inference performance for the Qwen-3.8-Flash-Next model in Apple Silicon environments. This branch supports n-gram tensor SSD offloading alongside Metal backend optimizations, enabling the execution of highly quantized models that exceed RAM and VRAM capacities.
Benchmark results on an M1 Ultra showed that for a 128B parameter Q8_0 quantized model, prompt processing (pp512) reached 530 tokens per second, and token generation (tg128) reached 31 tokens per second. Notably, the model maintained efficient inference with pp512 performance of 410–449 tokens per second even in long contexts (d10000–30000).
While the SSD offload feature causes significant performance degradation in prompt processing speed, token generation speed is relatively less affected, making it viable for practical use. This enables the deployment of large-scale models that surpass hardware memory limitations.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Jared Palmer Releases 'Kev', a Lightweight Decision-Making Model Based on Qwen3.5