AI BriefingTransformers Gradient Accumulation Fix
Key point
HuggingFace fixed an issue in the Transformers library where loss function calculation during Gradient Accumulation was mathematically inaccurate.
Details
In HuggingFace's Transformers library, it was discovered that when using Gradient Accumulation, a mathematically correct loss was not being calculated, and this has now been fixed.
Cause of the Problem For token-level tasks such as Causal LM, the correct loss value should be obtained by dividing the total loss across all accumulation steps by the total number of non-padding tokens. However, the existing approach averaged the loss values across each batch, causing a mathematical inconsistency.
Solution HuggingFace resolved this issue in two ways:
- Automatic Fix for the Default Loss Function: When users use the default loss function, the internal logic has been updated so that the correct loss is reported and utilized during Gradient Accumulation.
- Providing a Custom API: By exposing the
loss_functionAPI so users can directly pass their own loss function to theTrainer, this allows for flexible handling of similar issues that may arise in the future.
Fixes for major models are currently in progress, and users can immediately apply the fix by installing the main branch of transformers.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Microsoft's Full-bandwidth Transformer