NVIDIA Releases Model Lightweighting Tool

NVIDIA has released NVIDIA Model Optimizer to address memory and latency issues in LLM inference. This library integrates scattered optimization techniques such as quantization, pruning, and knowledge distillation into a unified framework, exporting results in a checkpoint format that major inference engines like vLLM, TensorRT-LLM, and SGLang can load immediately.

Key Optimization Techniques and Features

Model Optimizer offers six techniques depending on whether training is required.

• No Training Required: Post-Training Quantization (PTQ), Sparsity. Can be quickly applied on a few GPUs using only calibration data.
• Training Required: Quantization-Aware Training (QAT), Knowledge Distillation, Speculative Decoding, Pruning (with retraining). Integrates with Megatron-Bridge and Hugging Face Accelerate to construct training pipelines.

In a case study applying pruning and FP8 quantization to the Nemotron-3-Nano-30B-A3B model, vLLM throughput and memory usage improved by 2.6x respectively.

Quantization Method Selection Guide

In practice, the optimal quantization method should be selected based on batch size and GPU generation.

• FP8: Very low accuracy loss and compresses to 50% of the original size. Recommended for Ada/Hopper and later GPUs. NVIDIA presents this as the default recommendation.
• INT4 AWQ (W4A16): Advantageous for small-batch inference (batch size 4 or less). Compresses to 25% of the original size but has lower performance for large batches.
• INT4-FP8 AWQ (W4A8): Ensures performance in large-batch serving environments by quantizing both weights and activations.

Benchmark results show that for the Llama3.1-70B model at batch size 64, INT4 AWQ is 0.88x compared to the baseline, actually slower, indicating that unconditionally choosing lower bit depths can result in performance losses in serving environments. In contrast, FP8 showed a 2.10x speed improvement at batch size 64.