Qwen 35B MoE on an RTX 3090

For Windows, this outlines a procedure to run Qwen 3.6 35B MoE on an RTX 3090 24GB and 64GB RAM, loading the model into 12.4GB VRAM while maintaining 262K context. The author writes that this combination achieves about 111 tok/s.

The core is the llama.cpp-tq3 fork and TurboQuant. It compresses weights and the KV cache more efficiently than regular GGUF, reducing memory burden, and explains that MoE and DeltaNet + Gated Attention mitigate performance degradation at long context. Since only 10 attention layers out of 40 layers grow their cache, the KV cache is much smaller than a dense model's.

Installation proceeds in the following order.

• Prepare Git, CMake, Ninja, Visual Studio 2022 Build Tools, CUDA Toolkit on Windows.
• After conda create -n qwen-local python=3.11 -y, install huggingface_hub[cli].
• Download Qwen3.6-35B-A3B-TQ3_4S.gguf and mmproj-BF16.gguf from Hugging Face, and attach vision support via mmproj.
• Build llama.cpp-tq3 with CUDA enabled.
• In the .bat, set CONTEXT=262144, KEY_TYPE=q8_0, VAL_TYPE=q8_0, BATCH_SIZE=2048, REASONING=on, and run llama-server.exe with -fa on, --jinja, --reasoning, --reasoning-budget 2048 attached.

The default recommendation is q8_0/q8_0, and it writes that with this setting 262K context uses only about 2.7GB. It can be reduced further with tq3_0 or q4_0, but since server settings don't change while running, you must restart if you change the values.