StepFun Step 5 Preview Weights Released, Runnable on 8 H200s

• StepFun released the FP8, BF16, and NVFP4 weights for Step 5 Preview on Hugging Face and ModelScope on October 15
• Sparse MoE architecture with 600B total and 27B active parameters, supporting up to 1M token context and image input
• The FP8 checkpoint requires approximately 650GB of storage and can run on 8 H200s (1,128GB) or 4 B300s (1,152GB)
• Achieved a throughput of approximately 7,800 tokens per second per GPU using the NVFP4 checkpoint on NVIDIA GB300 NVL72 (under 8K input and 1K output conditions)

Technical Architecture and Optimization

• Adopted a 'Narrow but Deep' design with a hidden dimension of 4,096 and a 92-layer structure
• Applied the Sparse GQA mechanism, reducing indexer latency by 7.89 seconds and increasing attention latency by 0.15 seconds
• Used FP32 hidden states and RMSNorm scaling, along with fine-grained gradient clamping, to stabilize FP8 training

Serving and License

• Not included in the official vLLM v0.32.0 release; a dedicated Docker image is required
• Subject to the StepFun Model License 1.0; MaaS providers with annual revenue exceeding $20 million and hardware manufacturers require separate commercial agreements