llama.cpp adds support for DeepSeek v4 Flash

Experimental support for running DeepSeek v4 on llama.cpp has been released.

The released GGUF model can run inference with about 128GB RAM, and the author reported achieving around 17 tokens per second on a MacBook M3 Max.

The quantization approach was applied as follows.

• Routed expert weights are heavily quantized to 2bit
• Different 2bit quants are mixed to balance error and size
• Each layer's shared expert is kept at Q8

The author noted that response quality was quite good even under 2bit quantization, but added that whether it can outperform Qwen 3.6 27B still requires further experiments and benchmarks.