AI Briefing
KOSign in

NVIDIA Tutorial: Serving GLM-5.3 NVFP4 on Dual DGX Stations with vLLM

·2026.11.14 02:00

Key point

A technical guide details serving the 753B parameter GLM-5.3 NVFP4 model across two NVIDIA DGX Stations using vLLM, achieving a peak throughput of 607.8 tokens/sec.

1 / 3

Details

  • NVIDIA Technical Blog publishes a tutorial for serving the GLM-5.3 NVFP4 model (753B total parameters, 40B active) using vLLM on two GB300-based DGX Stations.
  • The setup utilizes Tensor Parallelism (TP=2) with ConnectX-8 RDMA, achieving a peak output throughput of 607.8 tokens/sec at concurrency 16.
  • Configuration requires vLLM v0.30.0-aarch64, FP8 KV-cache, and Multi-token prediction (MTP) with 4 draft tokens per step.
  • Memory requirements exceed a single GB300 GPU (252 GB HBM3e), necessitating distribution across two GPUs with approximately 231.35 GiB used per station.
  • Benchmarking with AIPerf on the SPEED-Bench Coding dataset shows throughput drops to 385.0 tokens/sec at concurrency 32 due to increased latency.
  • The deployment uses NCCL 2.30.7 to avoid CUDA graph replay deadlocks in RoCE environments and exposes a single OpenAI-compatible endpoint.
inferencevllmmodel-servingtensor-parallelismglm-5.3nvidia-dgx
Read originalShare on Threads

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.

Related

RedditMACaQ: Open-source quantizer with custom vLLM kernels and lower low-bit KLDRedditlibstrix Docker image enables Qwen3.8-Flash-Next on Strix Halo with 40+ tok/s decodeHacker NewsUbicloud Achieves 100% Tool-Call Accuracy in vLLM via Constrained DecodingPyTorchKRFlashStates Improves GLM-5.3-Flash Prefill Throughput by 3.14xPyTorchKRllama.cpp v1.0.0, First Major Release in Three and a Half YearsHacker Newsggml v1.0.0 Released with FP8 Support and Backend Optimizations