AI BriefingComputing Cross Entropy Loss While Saving Memory
Key point
Analyzes the memory spike problem that occurs during Cross Entropy calculation in LLM training and presents solutions.
Details
When processing long sequences such as 128K context during LLM training, we analyze the cause of sudden OOM (Out of Memory) occurring at the Cross Entropy Loss calculation stage.
The core of the problem lies in Logit Materialization. When using PyTorch's default F.cross_entropy, a massive Logit tensor gets materialized in memory during the calculation process, causing memory usage to spike vertically.
The key strategies to solve this are as follows:
- Introducing Memory-aware algorithms: An approach that processes only the parts needed for calculation, without loading the entire Logit into memory.
- Chunking technique: Suppresses memory peaks by dividing the sequence into smaller units (Chunks) for calculation.
- Utilizing Fused Kernels: Prevents the creation of large intermediate tensors by fusing operations into one.
These optimizations are essential for maximizing memory efficiency during Long Context training of large-scale models.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Jane Street Research Discovers Non-Monotonic Patterns in Sequence Weight Scaling
Naver Cloud Achieves 1.33x Speedup on Single Layer via Mamba-2 Kernel Optimization for B200 GPUs