AI BriefingRisk of Training Errors in AI-Generated CUDA Kernels
Key point
Research has revealed that AI-generated CUDA kernels can cause loss functions to diverge during actual model training due to precision issues.
Details
It has been revealed that AI-generated CUDA kernels that ranked at the top of NVIDIA's SOL-ExecBench can cause serious errors in actual production environments.
The key case cited was a fused embedding-gradient + RMSNorm backward pass kernel. Despite passing benchmark validation, when applied to actual transformer training, this kernel caused the loss to diverge, resulting in training failure.
The root cause was insufficient bf16 (bfloat16) precision.
- During the process of accumulating values in bf16 for embedding gradient computation, small gradient values from high-frequency tokens were pushed aside by larger values and rounded to zero.
- This caused data drift in the rows of certain tokens, breaking the training process.
This type of bug does not appear when the data distribution is uniform, and only surfaces in environments where specific tokens are concentrated, as in real text data—making it very easy for researchers to mistake it for a problem with the model itself.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Microsoft's Full-bandwidth Transformer