humans& Introduces GPU-Accelerated Weight Teleportation for RL Training

humans& released GPU-accelerated Weight Teleportation, a lossless GPU compression method that resolves weight synchronization bottlenecks in large-scale RL by outperforming intra-cluster P2P RDMA.
In Ampersand production RL configurations, the system achieves a sampler pause time of under 0.23 s and an end-to-end weight sync time of under 4 s, with the Blackwell Decompression Engine (DE) pipeline fully saturated.
The approach uses a double-compression strategy (Snappy + Zstd) to reduce the payload to 2.65 GB for NVFP4 GLM-5.3 models, compared to a 5.36 GB baseline for LZ4 alone.
Bitwise XOR delta updates and distributed GPU compression allow the system to handle sparse RL updates without disk round-trips, supporting use cases like fault tolerance and checkpoint resumption.

Technical Implementation

Delta updates rely on bitwise XOR operations to generate sparse masks, which are highly compressible due to the sparse nature of RL training signals.
A curated double-compression algorithm applies outer Zstd compression to GPU-friendly inner codecs (Snappy or LZ4) to minimize payload size while maintaining high decompression throughput.
The pipeline overlaps the fixed-function DE with SM compute using a double-buffered software pipeline, eliminating the need for full-model decompression output buffers in HBM.

The final "Teleporter" configuration achieves a trainer prep/compression time of 0.789 s and a receiver prepare time of 2.355 s for the 64-GPU trainer setup.
Low-precision RL compatibility is maintained by treating weights as contiguous bit buffers, allowing NVFP4 weights to absorb small high-precision updates with a 3.6x smaller storage footprint than BF16.
The system supports inference serving simplification by transmitting tiny compressed deltas instead of full models, with an opt-in release_state argument to free resources after delta application.