Achieved 207 tok/s with Qwen3.5-27B on an RTX 3090

Lucebox released two implementations that push up Qwen3.5-27B inference performance on an RTX 3090.

• Megakernel Qwen3.5 0.8B: a persistent kernel that packs 24 layers into a single CUDA dispatch, recording 37,800 on prefill pp520, 413 tok/s on decode tg128, and 1.87 tok/J. The comparison target, llama.cpp BF16, recorded 11,247 / 267 tok/s and 0.76 tok/J.
• DFlash DDTree Qwen3.5 27B GGUF: runs on a single RTX 3090 with a Q4_K_M target + BF16 draft and budget=22 setting, producing 207.6 tok/s in the demo (5.46x vs AR's 38.0 tok/s). The HumanEval average is presented as 129.5 tok/s, 3.43x vs AR, and 15% faster than chain speculative decoding.
• To fit 128K context within 24 GB as well, they used a Q4_0 KV cache and a sliding target_feat ring. In this configuration, it showed 134.78 tok/s at 131072 ctx.
• On the implementation side, they newly ported a C++/CUDA decode engine on top of ggml, and added 3 custom CUDA kernels for tree-aware SSM rollback: ggml_ssm_conv_tree, ggml_gated_delta_net_tree, ggml_gated_delta_net_tree_persist.

The key point is that this project isn't just impressions—it's a reproducible performance report that includes hardware, model, quantization, context length, and benchmark figures.