85TPS on a single 3090

To run Qwen3.6-27B on a single RTX 3090 24GB, quantization, speculative decoding, and KV cache compression were tuned step by step.

• With base AWQ INT4, performance was around 38 TPS / 8K context / 21.4GB VRAM,
• and quantized versions without an MTP head caused OOM at 24GB because vLLM newly allocated a BF16 MTP head of 2.37GB.
• Lorbus/Qwen3.6-27B-int4-AutoRound kept only mtp.fc in BF16, allowing vLLM to load MTP normally, achieving 54.6 TPS(narr) / 59.0 TPS(code) with an average accept length of 1.9.
• Increasing the number of speculative tokens showed the best efficiency at n=3, raising performance to 63.8 TPS(narr) / 79.7 TPS(code) and AL 3.4.
• With FP8 KV cache, context extended to 20K with almost no TPS loss.

The next bottleneck was TurboQuant KV cache. Due to the hybrid attention + DeltaNet architecture, vLLM blocked this by default, so an external patch bundle called genesis-vllm-patches was applied to bypass the hybrid gate.

Afterward, with the TurboQuant k8v4 setting, the KV pool grew significantly, but a crash occurred during CUDA graph capture in the warmup stage due to an issue with query_start_loc.tolist(). The workaround required disabling CUDA graph, and the final configuration settled at 85 TPS sustained / 106 TPS peak, 125K context, Vision enabled, 21.3 / 24GB VRAM, and a 230W cap.

Additionally, the initial GGUF file was found corrupted during aria2c download, and it was noted as a key lesson that the model only worked properly after re-downloading with SHA256 and HF linked-etag verification.