RL Training Speed Improved Up to 7.5x

Existing open-source RL engines repeatedly process the 'prompt + response' sequence for every sample in a group, which causes severe computational waste when prompts are long and responses are short.

To address this, Prompt Caching, a method that computes the prompt only once and then processes multiple responses, was applied to RL training. Considering that gradients must be passed to the prompt during training, technical optimizations were made to preserve the causal attention structure.

Test results based on the Qwen3.5-4B model are as follows:

• 16k prompt / 64 responses: 7.5x improvement
• 16k prompt / 128 responses: 7.3x improvement
• 16k prompt / 1k responses: 5.4x improvement
• 8k prompt / 4k responses: 1.7x improvement