TRL v1.14 Adds LoRA Async GRPO Support Without NCCL

The AsyncGRPOTrainer in TRL v1.14 supports LoRA adapter training and introduces a new architecture that synchronizes adapters between the Trainer and vLLM without NCCL in the Hugging Face Jobs environment. Thinking Machines' 'LoRA Without Regret' research demonstrates that rank-1 LoRA can achieve performance similar to full fine-tuning in RL; based on this, the system stores MB-sized adapters in storage buckets for vLLM to load. The proxy adds authentication headers, routes rollouts to replicas holding KV prefixes, and broadcasts adapter loads to all replicas.

Performance Optimization and Results

Experiments with the Qwen2.5-Math-1.5B model showed that wall clock time for 500 steps was reduced from 3 hours 27 minutes to 53 minutes, and MFU increased from 3.9% to 23.5%. Reward rose from 0.145 in the first 20 steps to 0.416 in the last 20 steps (based on run 5), validating effectiveness.

• Synchronization Speed: Weight synchronization time (p50) reduced from 8.5 seconds to 6.2 seconds
• Step Time: Time per step (p50) reduced from 22.9 seconds to 4.8 seconds