AI BriefingFrom scratch with a 5080
Key point
Trained a 235M-parameter LLM from scratch using a single RTX 5080.
Details
Trained a 235M-parameter LLM from scratch in PyTorch without pretrained weights.
The architecture consists of 18 layers, hidden size 1024, GQA 16 query heads / 4 KV heads, SwiGLU FFN 2816, RoPE (theta 10000), RMSNorm pre-norm, tied embeddings, and uses a 32k SentencePiece BPE vocab.
Training was fit onto a single consumer GPU, the RTX 5080, using bf16 mixed precision and gradient checkpointing, training on about 5B tokens at seq len 1024.
The data pipeline was also built from scratch.
- Used FineWeb-Edu, Wikipedia, StackExchange, code, ArXiv
- Quality and toxicity filtering
- MinHash deduplication
- Custom SentencePiece tokenizer
- Domain-weighted mixing
- Pretraining plus instruction tuning, with loss masking applied so only assistant tokens are trained on
Sample outputs were also shared. In the examples, it answered 1914 for a "World War 1" question, and gave a reasonable response to the definition of steak.
The next version, Plasma 1.1, is currently being trained at 500M parameters, aiming for better multi-turn conversation, a larger vocab, and byte fallback.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
LLM Mathematical Excellence Driven by Pre-training Data Quality, Not Verifiability