AI BriefingQwen3.6 35B Speed Test
Key point
On a single H100, Qwen3.6 35B hit 250-400 tok/s.
Details
Serving Qwen3.6 35B FP8 on a single H100 with SGLang, the author added an OpenAI-compatible API and Anthropic-compatible routing for Claude Code.
Performance was pushed up through multiple configuration sweeps.
- Compared decode settings, cache settings, speculative decoding variants, and backend combinations.
- The combination with the biggest impact was dflash speculative decoding with a matched draft model.
- FP8 weights and FP8 KV cache reduced memory pressure.
- Prefix caching, faster attention, and the MoE backend also made a meaningful difference on H100.
- Prefill and decode required different tuning.
Peak performance was 250+ tok/s for general sentence generation and 400+ tok/s for code-style generation. The author noted there's still room for deeper speculative decoding exploration and kernel optimization, but the current version stops here.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Qwen Intelligence Launches Three Mobile AI Agents and Open Benchmark Suite
Qwen3.5-based Small Decision Model 'Kev' Released with TypeSafe API Compatibility and Performance MetricsApple Releases LensVLM-9B for Processing Compressed Text Images