AI BriefingDeep Learning Optimizer Discovered by Genetic Algorithm
Key point
A genetic algorithm discovered an optimizer that outperforms Adam by 2.6%.
Details
A framework was presented that uses a genetic algorithm to automatically search for deep learning optimizers.
The optimizer encoded the following combination of primitive updates as a genome.
- gradient
- momentum
- RMS normalization
- Adam-style adaptive terms
- sign-based updates Hyperparameters and schedule options were also searched together.
After running multiple vision tasks for 50 generations with a population of 50, the evolved optimizer achieved a 2.6% improvement in overall fitness compared to Adam. On CIFAR-10, it achieved a 7.7% relative improvement.
The discovered design combined a sign-based gradient term with adaptive moment estimation, and used lower momentum coefficients than Adam (β1=0.86, β2=0.94). It also turned off bias correction and enabled learning rate warmup and cosine decay.
The researchers showed that evolutionary search can discover design principles different from hand-crafted optimizers, and released the code.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Sakana AI Releases 'Augmented Lagrangian Predictive Coding' Capable of Training 1000-Layer Neural Networks