Making LLM Training Faster with Unsloth and NVIDIA

Unsloth announced that it worked with NVIDIA to optimize the LLM training pipeline, pushing overall speed up by about 25% more. This improvement is applied without any loss of accuracy, and it stacks on top of Unsloth's existing 2-5x speedup. Updating to the latest version automatically activates this on RTX laptops, data center GPUs, and DGX Spark.

There are three key improvements.

• Caching packed sequence metadata instead of regenerating it for every layer, yielding a 14.3% speedup.
• Double-buffered async gradient checkpointing, improving performance by 8%.
• Using argsort and bincount in gpt-oss's MoE routing, boosting training speed by 15%.

In packed batches, the length, cu_seqlens, maximum sequence length, and attention structure are identical across multiple layers. By generating this information only once and reusing it, repeated CPU-GPU synchronization and preparation costs can be reduced, which lowers the bottleneck in the forward path.