Serving GPT-OSS 120B on RNGD at 5.8ms TPOT

At the opening event of OpenAI's Korea office, gpt-oss-120b was run on two RNGD cards, achieving 5.8ms TPOT and power efficiency of under 180W per card within weeks of launch.

The key was fitting the new MXFP4 format onto the existing hardware pipeline. FP4 pair → FP8 conversion, FP8 → FP32 cast, and scale factor application were all pipelined so that memory fetch and computation overlapped, resulting in good overlap with MLP weight loading and reducing the TPOT contribution.

Compiler optimizations were also added to efficiently handle even small tensors. The DRAM layout of the MXFP4 scale tensor and bias tensor was adjusted, and operations like attention sink were supported without data movement.

The MoE bottleneck was reduced with a gpt-fast style kernel. Since only 4 of 128 experts are activated per token, sparsely loading only the active experts was advantageous for latency, and the following optimizations were applied.

• Minimized kernel-launch overhead on the indirect indexing path
• Selected schedules closer to reality via HBM bandwidth modeling that accounts for runtime addresses
• Jointly optimized data layout and matmul to reduce the cost of gather → rearrange → matmul

For the multi-chip configuration, Tensor Parallelism (TP) was chosen. gpt-oss-120b requires at least 60GB when parameters, KV cache, and activations are combined, and with two RNGD cards (96GB total), TP could meet the memory requirement while also utilizing the HBM bandwidth of both chips.

Since this demo targeted single-user latency, TP was prioritized over EP. Data was tightly packed right before reduce-scatter to improve link efficiency, and during decoding, computation and data movement were merged into one large schedule across layer boundaries so that communication was hidden behind compute.

Ultimately, this performance was the result of naturally integrating MXFP4 into the hardware pipeline and redesigning MoE's dynamic access from a data-movement perspective. The authors stated they plan to further improve concurrency, EP, and TTFT to meet low latency across more service conditions.