Implementing MLM and CLM with PyTorch: Principles of Training Transformer Encoder-Based Language Models

This article explains the principles of MLM (Masked Language Model) and CLM (Causal Language Model), which are pre-training mechanisms for large language models, and covers the process of implementing them directly in PyTorch. While pre-trained models are highly versatile, they have large parameter counts and consume significant resources; building a model of appropriate size for a specific domain can maintain or improve performance while increasing training and inference speed.

The basic structure of the model is a Transformer encoder composed of an Embedding Layer, Multi-Head Self-Attention Layer, and Feed-Forward Layer. Comparing the key hyperparameters of BERT and GPT2, the vocab_size is 30,522 for BERT and 50,257 for GPT2, while max_position_embeddings are 512 and 1,024, respectively. MLM is trained using a Denoising Autoencoder approach that masks 15% of the sequence (80% , 10% random, 10% kept) to restore the original tokens. CLM uses a causal approach to predict the next token based on previous tokens, utilizing an Attention Mask to prevent referencing future tokens. Building such domain-specific models reduces resources and enhances experimental productivity.