AI BriefingQwen3.6-27B throughput doubled
·2026.04.28 01:31
Key point
Luce DFlash boosted Qwen3.6-27B throughput by up to 2x on a single RTX 3090.
Details
Luce DFlash is a ggml-based C++/CUDA stack that runs Qwen3.6-27B on a single 24GB RTX 3090.
- On HumanEval / GSM8K / Math500 benchmarks, it delivered an average 1.98x throughput compared to autoregressive decoding.
- It applied speculative decoding without retraining, and the corresponding Qwen3.6-DFlash draft is an in-training model released by z-lab on 2026-04-26.
- The target weights are loaded as Q4_K_M GGUF at about 16GB, and the draft as bf16 at about 3.46GB.
- The KV cache is compressed with TQ3_0 down to about 3.5 bpv, and is designed to fit a 256K context within 24GB using a 4096-slot ring.
- For prompts exceeding 2048 tokens, the prefill ubatch is automatically expanded from 16 to 192, delivering about 913 tok/s prefill on long prompts.
- Execution involves building with
cmake, downloading the GGUF model and draft, then running viascripts/run.py, and it is described as working without llama.cpp / vLLM / SGLang.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Optimizing Gemma 4 Serving with vLLM on Amazon EKS [Part 2: Throughput and Latency SLOs on a Single GPU]