AI BriefingQwen3.6-35B Applies MTP, Achieving 1.49x Inference Speed Boost
Key point
By applying MTP to the Qwen3.6-35B-A3B model, a decoding speed improvement of about 1.49x was achieved on llama-server.
Details
The Hi-Fi MTP runtime version of the Qwen3.6-35B-A3B model has been released. This model includes an MTP (Multi-Token Prediction) draft head, enabling native Speculative Decoding support in llama-server.
In tests conducted on an A100-80GB GPU environment, a 75.99% draft token acceptance rate was recorded, confirming an approximately 1.49x decoding speed improvement over the conventional approach. (Baseline 133.6 tok/s $\rightarrow$ approximately 199 tok/s with MTP applied)
The key performance metrics and features are as follows:
- Chat Template Usage: Optimal performance (acceptance rate ~76%, 1.49x speed) appears when using the
/v1/chat/completionsendpoint with the Qwen chat template. When using raw prompts without the template, the acceptance rate drops to 63%. - Draft Window ($k$) Optimization: Test results showed the best performance at $k=4$, with acceptance rates dropping sharply as $k$ increases due to accumulated prediction errors in the draft head.
- Domain-specific Acceptance Rates: Performance was consistent across domains, with Code (73.7%), Math (78.1%), and General (76.6%).
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
vLLM Publishes Performance Benchmarks for 5 Speculative Decoding Methods on AMD GPUs
3.85x Inference Speedup via Diffusion Draft Generation and AR Verification Based on GLM-OCRApple Releases LensVLM-9B for Processing Compressed Text Images
Qwen Releases 'RecreationWorld' for Training AI Agents to Rebuild Apps