Encoder-Decoder MT Dynamic Batching

The author reported that when fine-tuning NLLB-200 600M on an RTX 5090, a fixed batch size of 8 was the OOM limit, and a few long samples would bundle the whole batch together, lowering GPU utilization.

dynabatch sorts examples from longest to shortest by token length, uses the longest batch as a reference point, and increases the candidate batch size for shorter batches. It then uses an XGB regressor to predict memory pressure and selects the maximum batch that stays under the safety threshold.

The key points are as follows.

• Uses source length as a proxy for target length in encoder-decoder MT
• Achieved a 3.3x throughput improvement over fixed batching in training benchmarks
• On Colab T4 generation benchmarks, the improvement was around 1.06x to 1.21x
• Added a fallback path to account for regressor error and OOM

The author noted that sequence packing is better suited for decoder-only models, and presented this method as a niche batch-optimization tool focused on MT.