AI BriefingModel Architectures in Code
Key point
An open-source book and repository implementing everything from GPT-2 to Llama 3.2-3B to DeepSeek in code.
Details
A book and open-source code have been released that implement modern LLM architecture from scratch, based on GPT-2.
In Chapter 3, only 4 elements are changed from GPT-2 to expand it into the Llama 3.2-3B form.
- LayerNorm → RMSNorm
- Learned positional encodings → RoPE
- GELU → SwiGLU
- Multi-Head Attention → Grouped-Query Attention
It doesn't stop there — it even loads Meta's actual pretrained weights to align the behavior.
In Chapter 5, the full DeepSeek architecture is built.
- MLA and the absorption trick
- decoupled RoPE
- MoE with shared experts and fine-grained segmentation
- auxiliary-loss-free load balancing
- Multi-Token Prediction
- FP8 quantisation
The code is available on GitHub, and the book offers a free sample on Leanpub. It's a practical resource for anyone who wants to understand model internals at the code level.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Jevper Released: A Jev Classification Wrapper for OpenAI-Compatible Models with Automatic Logprobs Detection