Tang Project Derives GPU Kernels via IR, Cutting 80B MoE Decode Step Time by 5-12% at High Concurrency on H200

Details

The Tang project replaces manual kernel writing with an IR-based compiler that derives schedules using rewrites like monoid reductions and affine maps, generating kernels for various hardware targets.
On H200 GPUs, the derived approach reduced decode step time by 5-12% for an 80B MoE model (Qwen3-Coder-Next) at 16 to 48 concurrent streams, though it remains slower than hand-written kernels at lower batch sizes (1 and 4).
Prefill performance improved, reducing Time to First Token (TTFT) for 44k tokens from 8.33s to 5.54s (a one-third reduction), although vLLM remains approximately 12x faster than the engine's baseline.
Derived attention kernels are not uniformly fast; on RTX 3090, they were up to 150x slower than FlashAttention-2 for causal forward passes, and on H200, they lagged FlashAttention-3 in both forward and backward passes.
The system achieves 148.7 tok/s decode speed for Qwen3-8B, matching llama.cpp, but the derived path missed a 99% accuracy gate at 48 streams (scoring 98.93%), an issue linked to RMSNorm numerical instability that was addressed by increasing split granularity.
Speed-of-light analysis using Brascamp-Lieb inequality shows derived kernels often fail to reach theoretical time floors, with H200 median floor reach rate at 0.38.
The system lowers one graph to multiple targets including WGSL, CUDA, Metal, OpenCL, GLSL, and C, supporting hardware from datacenter GPUs to phone-class SoCs.

Key point