AI BriefingToken AI Announces New Optimizer STAM
Key point
Token AI has announced STAM, a new optimizer that enhances training stability and maximizes memory efficiency.
Details
STAM (Stable Training with Adaptive Momentum), announced by Token AI, is a new algorithm designed to overcome the limitations of existing optimizers such as Adam and AdamW.
Existing Adam-family optimizers use a fixed beta1 value, which causes an issue where past momentum leads the model in the wrong direction when noise occurs or the direction changes abruptly during training. STAM measures the difference between the current gradient and the previous momentum ($g - m$) and the residual variance to dynamically adjust beta1, thereby enhancing training stability and self-correcting errors.
In particular, the lightweight version STAMLite has been presented as an efficient alternative that can replace AdamW.
- Memory efficiency: While AdamW uses about 2x the parameter size, STAMLite requires only about 1x the optimizer state memory.
- Resource savings: This allows for approximately 50% resource savings compared to the existing method.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
JiT-DDT Architecture Improves Text-Image Model Training Speed by 3.6x Compared to Linum v2