AI BriefingDispersion loss offsets embedding condensation in small language models
Key point
We propose a new training objective to address the embedding condensation phenomenon that occurs in small language models.
Details
We discovered a phenomenon called embedding condensation, in which token embedding vectors converge into a narrow cone-shaped subspace as they pass through Transformer layers. This phenomenon appears more severely as model size decreases, causing the vectors to point in nearly parallel directions.
According to the research findings:
- Correlation with model size: Within the same model family, this condensation phenomenon decreases as model scale increases, with larger models being more resistant to it.
- Reproducibility: The same phenomenon was observed in controlled experiments where only the MLP dimension was adjusted, confirming a close relationship with model size.
- Onset timing: This phenomenon appears from the model initialization stage, and while it is gradually mitigated through pre-training, it does not disappear completely.
To address this geometric issue, the researchers designed a new training objective called dispersion loss, which increases the variance of embeddings, thereby improving the model's representational capacity.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.