AI BriefingImplementing a Transformer in C++17
Key point
An 826,985-parameter Transformer built in C++17 achieved a validation loss of 1.6371 after training for 76.2 minutes on a single CPU core.
Details
Quadtrix.cpp is a GPT-style decoder-only language model built using only the C++17 standard library and POSIX sockets. It uses no PyTorch, LibTorch, BLAS, or autodiff libraries.
Implementation Scope
- tensor library, token/position embeddings, LayerNorm, Linear, Dropout
- multi-head causal self-attention, feed-forward block, character-level tokenizer
- backward pass for cross-entropy, softmax, layernorm, attention, Q/K/V, ReLU, dropout, embedding scatter-add
- AdamW optimizer and OpenMP parallelization
Training Setup
- 4 layers x 4 heads x 200d decoder-only transformer
- 826,985 params
- 128-character context window
- 31.4M characters of children's story corpus
Results
- 76.2 minutes of training on a single CPU core
- Minimum validation loss of 1.6371 nats
- With OpenMP applied, 5-7x speedup on 8 cores for matmul, bmm, softmax, and layernorm
For LayerNorm backward, mu, inverse-std, and x-hat were stored during the forward pass to apply the 3-term formula, and for attention backward, the dropout masks applied to attention weights and projection output were tracked separately.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.