AI Briefing100 tps on RTX 5090
Key point
Qwen3.6-27B-INT4 achieved 256k context and 105~108 tps on a single RTX 5090.
Details
Running Qwen3.6-27B-INT4 on 1x RTX 5090 with vllm 0.19 achieved 105~108 tps. This figure was measured while maintaining the native 256k context.
The key factors are the lightweight Lorbus/Qwen3.6-27B-int4-AutoRound quantized model and MTP-based speculative decoding. The author explained that because the model size is reduced, the maximum context length can be used as-is without a separate TQ.
The execution settings are as follows.
--max-model-len 262144--attention-backend flashinfer--kv-cache-dtype fp8_e4m3--max-num-seqs 2--enable-prefix-caching--enable-chunked-prefill--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
The author stated that this configuration produced faster results than the previous day, and that the local inference efficiency of the Qwen3.6-27B series continues to improve.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Ollaya Launches Local Runtime for Open-Source Decision Models with TypeSafe SDK CompatibilityStanford and NVIDIA Release 'Contrastive Language Model' CLM with 9x Faster Inference
Meta Introduces Muse Realtime Avatar for Synchronized, Expressive Video Generation