PyTorch GPU Memory Visualization Guide

This guide explains how to identify and resolve the causes of RuntimeError: CUDA out of memory errors by using PyTorch's built-in features to visualize GPU memory usage.

Using PyTorch Memory Visualization Tools

• Use torch.cuda.memory._record_memory_history() to start recording a memory snapshot.
• You can upload the profile.pkl file generated by torch.cuda.memory._dump_snapshot() to the PyTorch memory viz website to view a visualized graph.

Memory Profiling Analysis

• Tracks step-by-step memory changes that occur during model creation, input tensor allocation, and the forward pass.
• In particular, it explains in detail how the model retains activation tensors for backpropagation, and how memory is freed when variables are reassigned.
• Using a real-world LLM training loop with Qwen2.5-1.5B as an example, it provides a practical method for profiling the memory patterns that occur during the training process of complex models.