AI BriefingMirai's uzu achieves 142.3 tok/s speculative decoding with Qwen3.6 27B on M5 Max
·2026.11.01 10:30
Key point
uzu, a Rust inference engine for Apple Silicon, recorded a speed of 142.3 tok/s on the M5 Max.
Details
- London-based Mirai unveiled uzu, a Rust-based local LLM inference engine for Apple Silicon, and presented benchmarks based on the M5 Max
- In an M5 Max 128GB environment, the Qwen3.6 27B model recorded performance of 142.3 tok/s for speculative decoding, 35.6 tok/s for autoregressive decoding, and 1,665 tok/s for input processing
- Speculative decoding performance is 2.2 times that of oMLX and 3.0 times that of llama.cpp, while input speed on M5/M6 chips is 1.5 to 1.6 times faster than MLX based on the median
- Provides Swift, Python, and TypeScript bindings and supports an OpenAI-compatible HTTP server CLI, available in macOS/iOS 26.4 or higher environments
- Uses Mirai's proprietary quantization formats (Mirai-M, Mirai-L), requiring conversion of public Hugging Face models via the lalamo tool
- Does not support Linux/Windows and supports only about 30 models from the Qwen, LFM, and Muse-Glimmer families; direct comparison with the original model is recommended for evaluating Korean quality
Technical Features and Limitations
- Applies speculative decoding with a DFlash-Weaver hybrid structure, optimizing the int8 path for the M5 GPU Neural Accelerator
- Most high-level API crates are classified as legacy, requiring version pinning
- Provides 4-bit asymmetric integer quantization (Mirai-M) and 8-bit symmetric quantization (Mirai-L), applying a block size 32 random Hadamard transform
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
OpenAI releases GPT-6.1 Terra with 50% lower API costs and improved computer useProspero: A lightweight C++ coding harness for local LLM agents