Catalyst Achieves Hyperparameter Optimization with a Single Differentiation

The Catalyst project announced that it achieved an average 27.8% reduction in validation loss across all 5 unseen seeds by applying a single bounded Catalyst-guided update to 50 AI training control variables (hyperparameters).

Experimental Setup and Methodology

• Target: Small pre-trained networks with rank-2 LoRA adapters
• Variables: 50 continuous control variables, including learning rate schedule, Adam betas, and weight decay
• Validation: Differentiation calculated using 3 development seeds, followed by final testing on 5 unseen seeds (10–14)
• Hardware: 1 core of Intel i5-11400H CPU, no GPU used, utilizing Rust-based compiled functions

Key Results

• Performance Improvement: Held-out mean validation loss decreased from 0.00958 to 0.00692
• Consistency: Improvement confirmed across all 5 unseen seeds (maximum -38.0%, minimum -10.7%)
• Sensitivity Analysis: beta2, ema_decay, and beta1 showed the highest influence, in that order
• Verification: Derivative accuracy confirmed through 101 verification points

Significance and Limitations

• Significance: Ensures generalizability through multi-seed-based differentiation rather than a single stochastic trajectory, reducing trial-and-error in AI engineering
• Limitations: Measures local sensitivity rather than discovering global optima; scalability to large models remains unverified
• Comparison: Unlike standard best-configuration search, it provides measured information on the directionality and relative importance of control variables that are not directly revealed