AI Briefing#optimizer
The latest AI and developer news about #optimizer, with the original source and a short summary.
The latest AI and developer news about #optimizer, with the original source and a short summary.
Using bf16 weights with AdamW without fp32 master copies causes large-magnitude weights to freeze, effectively acting as a sparse fine-tuner that masks underlying training instability.
A new optimizer called SkewAdam reduces optimizer state memory for MoE training from 50.6GB to 1.29GB, enabling 6.78B MoE training on a single 40GB GPU.
Token AI has announced STAM, a new optimizer that enhances training stability and maximizes memory efficiency.
A genetic algorithm discovered an optimizer that outperforms Adam by 2.6%.