Talkie: 1930's 13B vintage language model

Released talkie-1930-13b-base, trained on 260B tokens of English text from before 1931, and talkie-1930-13b-it, built without any modern chat transcripts or instruction-tuning data.

This project is an attempt to see how temporal knowledge cutoffs affect a model's generalization, creativity, and bias, through a 'vintage LM' built purely on historical text. The authors also run talkie as a live feed where Claude Sonnet 4.6 asks it questions and draws out answers in real time.

Key evaluation results are as follows.

• At 13B scale, talkie is the largest vintage LM the authors are aware of.
• It generally scored lower than its modern twin of the same architecture (trained on FineWeb) on standard knowledge evaluations.
• However, filtering out anachronistic questions cut the gap by nearly half.
• On HumanEval-style Python tasks, the vintage model is still weak, but on some problems it produced answers correct down to a single-character fix.
• Historical text transcribed with traditional OCR showed training efficiency of about 30% compared to human transcripts, which recovered to 70% with regex cleanup.

For post-training, instead of modern chat data, they started with structured text drawn from historical documents.

• They built instruction-response pairs from etiquette manuals, letter-writing manuals, cookbooks, dictionaries, encyclopedias, and poetry/fable collections.
• They applied synthetic prompts and online DPO, and under a Claude Sonnet 4.6 judge, the instruction-following score rose from 2.0 → 3.4/5.

The authors stated they plan to strengthen leakage detection further, improve vintage OCR, and scale up to GPT-3-class models and historical corpora of over 1 trillion tokens.