Transferring 1 Trillion Parameters via Hub Bucket: TRL's Delta Weight Sync

In Async RL, there is a problem where the entire model must be transferred to the inference engine at every step. A 7B model requires transferring about 14GB of data each time, and a 1T-scale model requires up to 1TB, consuming massive bandwidth and causing GPU idle time.

Delta Weight Sync, introduced into the TRL library, takes advantage of the fact that about 98-99% of the weights are identical between consecutive RL steps. Only the changed portions are made into sparse safetensors files and uploaded to a Hugging Face Bucket, which vLLM then fetches.

Through this technology, the per-step payload for the Qwen3-0.6B model was drastically reduced from 1.2GB to 20-35MB. This enables efficient distributed training via a shared store even when the trainer and inference engine are on different clusters, without a direct network connection (RDMA, VPN, etc.).