CS336: Language Modeling from Scratch

Stanford University's CS336 is a hands-on curriculum aimed at implementing language models—the foundation of modern NLP—from scratch. It covers the entire pipeline, from data collection and curation to building and training Transformers, and evaluation before deployment.

The main assignments and scope of study are as follows:

• Basics: Implementing tokenizers, Transformer architecture, optimizers, and training a basic model.
• Systems: Model profiling, optimizing FlashAttention2 using Triton, and implementing memory-efficient distributed training.
• Scaling: Understanding the function of model components and predicting model scaling through Scaling Law.
• Data: Building pretraining data through filtering and deduplication using Common Crawl data.
• Alignment & Reasoning: Implementing reasoning and safety alignment through SFT (Supervised Fine-Tuning), RL (Reinforcement Learning), and DPO (Direct Preference Optimization).

To take the course, proficiency in Python and PyTorch is required, as well as system concepts such as memory hierarchy, and mathematical foundations including linear algebra and probability/statistics. For large-scale GPU computation, using cloud GPUs such as Modal, Lambda Labs, or RunPod is recommended.