Ant Group and vLLM Community Open-Source Humming for High-Performance Quantized Inference

• Ant Group and the vLLM community released Humming, an open-source kernel library for quantized inference supporting NVIDIA Turing and later architectures as well as T-Head Zhenwu 810E.
• End-to-end inference on DeepSeek-V4.1-Flash with 8x H20 GPUs improved throughput by 14% to 43% and reduced Time To First Token by 25% to 32% when switching from Marlin W4A16 to Humming W4A8.
• Humming outperforms vLLM Marlin, FlashInfer, and DeepGEMM in most representative workloads, with specific gains including 2.6x speedup over DeepGEMM for W8A8 on H20 and 4.5x for W4A8 on H200.
• The library uses a general-purpose architecture with unified quantization representation and JIT compilation, reducing kernel compilation time to 1.82 seconds compared to 23.35 seconds for CUTLASS + NVCC.

Architecture and Performance

• Unified quantization representation converts various export formats into a standard Humming format, preserving quantization semantics while parameterizing weight formats, scaling grains, and data types.
• JIT compilation uses NVRTC or NVCC to compile kernels on-demand, with caching for reuse; a single layer GEMM kernel compilation and loading takes approximately 2 to 4 seconds.
• Supports native microscaling formats (MXFP8, MXFP4) on Blackwell Tensor Cores, enabling W8A8, W4A8, and W4A4 within a single kernel framework.

Integration and Availability

• Installable via pip install humming-kernels as a lightweight wheel (~500 KB) requiring PyTorch 2.7+.
• Integrated into vLLM 0.29.0+ as a dependency for CUDA installations, with options for --linear-backend humming and --moe-backend humming.
• Future plans include support for Rubin architecture, T-Head PPU, and autotuning for specific workloads.