Padding-Minimizing Efficiency

In LLM training, padding is still a major source of waste. Especially in the policy update step of online-RL, both the reference forward pass and the training forward-backward pass must handle variable-length sequences, so padding cost easily grows due to long-tail distributions and variation in response length.

Transformers can reduce this problem with sequence packing, but the same optimization is not straightforward to apply to state-space families like Mamba or hybrid Transformer-SSM architectures. This limitation showed up in the combination of Jamba2-3B and VeRL, and padding truncation alone still left length variance within micro-batches, falling short of the efficiency achieved by sequence packing.

The key consists of two steps.

• micro-batch-level truncation: shift left-padding to the right so padding is gathered on one side, then trim the padding at the end of the micro-batch.
• padding-aware dynamic micro-batching: sort by length and reconstruct micro-batches so shorter sequences aren't excessively matched to longer ones.

Experiments were conducted on GSM8K, on a single H100 node (8 GPUs), with batch size 32, max prompt 4k / max response 8k conditions. With padding truncation alone, policy update step time dropped from Jamba2-3B 45s→20s and Qwen2.5-7B-Base 63s→22s, while non-padding token output stayed the same and downstream quality was maintained. Overall, about 90% of padding-related overhead was eliminated, demonstrating a broader optimization path that doesn't depend on model architecture.