Soul Player C64 – a real transformer running on a 1MHz Commodore 64

Soul Player C64 is a 2-layer decoder-only transformer that runs on an unmodified Commodore 64.

• It uses about 25,000 int8 parameters, structured with 4 attention heads × 8 dimensions, 32-dimensional embedding, and a 64 hidden-unit FFN.
• Real multi-head causal self-attention, real softmax, and real RMSNorm are implemented in 6502/6510 hand-written assembly.
• Performance is around about 60 seconds per token, and the entire model is small enough to fit on a floppy disk.
• The key implementation point is a fix to integer attention score normalization. The score is adjusted to shift by only 14 bits, allowing the 128-entry exp lookup table to produce meaningful attention weights.
• The model is trained with QAT (quantization-aware training), evaluating actual int8 forward pass results during training.
• The training pipeline builds a BPE tokenizer (128 tokens), saves checkpoints, and then builds them into .prg and .d64 disk images.
• Testing verifies the full path from float reference → integer reference → memory-faithful shadow → 6502/6510 assembly → build round-trip.

Instructions for running it are also provided. On an emulator (VICE), you can start it with LOAD"SOULPLAYER",8,1 followed by RUN, and locally you can run the same integer operations faster with python soulchat.py.