Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings

For most languages, language model pre-training operates in a data-constrained regime where models must repeat training data many times, degrading generalization. A new study compares two remedies: aggressive hyperparameter tuning and mixing in data from a high-resource auxiliary language. The research finds that bilingual mixing yields larger improvements than hyperparameter tuning on both validation loss and downstream task accuracy.

Key Findings

The study systematically compared these approaches across four model scales from 150M to 1.43B parameters, using Arabic as the low-resource target and English as the auxiliary language. The analysis covered approximately 1000 pre-training runs.

• Superior Performance: Mixing outperforms hyperparameter tuning, with the performance gap growing as model size increases.
• Quantified Gains: Mixing boosts performance by an amount equivalent to 2–3× the unique target data on validation loss and 2–13× on downstream task accuracy.
• Validation Loss Limitation: Target-language validation loss systematically underestimates the value of mixing because it captures only regularization effects, not the new knowledge contributed by the auxiliary data.

Practical Recommendations

The authors recommend prioritizing the mixing ratio over hyperparameter tuning for data-constrained settings. They also suggest transferring hyperparameters from a small proxy model via μP (Maximal Update Parametrization) to handle scaling efficiently. This approach leverages the diversification of the training signal provided by auxiliary data, which repeated target corpora cannot supply.