AI BriefingTandem-vLLM 0.2.0 Enables DeepSeek-V4.1-Flash on Ampere, Ada, and Hopper GPUs
·2026.11.14 06:03
Key point
The release expands hardware support to include two-GPU setups and achieves 75.5 tok/s on a single RTX PRO 6000 card.
Details
- Tandem-vLLM 0.2.0 extends DeepSeek-V4.1-Flash support from Blackwell-only to Ampere (RTX 30xx/A6000), Ada (RTX 40xx/6000 Ada), and Hopper (H100/H200) GPUs.
- The engine uses 2-bit vector quantization for routed experts in host RAM, with a GPU cache for hot experts and CPU fallback for cache misses to ensure exact output.
- Two-GPU configurations are supported via tensor parallelism or a helper GPU mode, which offloads expert computation to a second card to reduce CPU load.
- Performance benchmarks show 75.5 tok/s on a single RTX PRO 6000 with a 64-core CPU, while 24 GB cards achieve 16.8-21.7 tok/s depending on the configuration.
- Quality tests indicate the 2-bit experts reduce GPQA-diamond accuracy to 76.8% compared to 81.3% for the official FP4 checkpoint, though GSM8K and long-context retrieval remain unaffected.
- Deployment requires 300 GB of disk space, 128 GB+ of RAM, Linux, the CUDA 13 toolkit, and an AVX2 or AVX-512 CPU.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
NVIDIA Tutorial: Serving GLM-5.3 NVFP4 on Dual DGX Stations with vLLM
PyTorch adds compile-time heuristics to Helion for faster kernel defaultsHuawei releases MACaQ, a model-agnostic quantizer for local inference