Talkie, a 13B Vintage Language Model from 1930

A 13B language model called Talkie, trained on 260B tokens of pre-1931 English text, has been released. It was designed to run conversation and generalization experiments without knowledge of the modern world, and a comparison model with the same architecture trained on modern web data, talkie-web-13b-base, was also built alongside it.

On standard language model evaluations, Talkie scored lower on average than the modern model, but the gap narrowed significantly once anachronistic questions were filtered out. In particular, comparable performance was confirmed on language understanding and mathematical tasks, and on the code generalization test HumanEval, performance slowly improved with scale.

The core contribution is an evaluation environment "fixed at a past point in time."

• It measured surprise beyond the knowledge cutoff to test the model's ability to predict future events and arrive at new ideas.
• Because it's an environment with less data contamination, pretraining generalization can be observed more directly.
• Although only pre-1930 text was intended to be used, temporal leakage actually remained, mixing in some knowledge related to Roosevelt, the New Deal, and World War II.

The challenges in the data and training pipeline were also addressed in detail.

• The corpus was assembled from books, newspapers, periodicals, scientific journals, patents, and legal case records.
• Historical documents from the 1930s had low OCR quality, and training on existing OCR transcripts yielded only 30% of human-transcript performance at the same compute.
• Simple regex cleaning recovered performance to 70%, but the gap remained large, and vintage OCR re-transcription is planned to further close it.

Post-training was also newly constructed without modern instruction data.

• Instruction-response pairs were created from etiquette books, letter-writing guides, cookbooks, dictionaries, encyclopedias, poetry collections, and fable collections.
• Synthetic prompts were used to generate summaries, direct information requests, and multi-turn conversations, and DPO was applied with Claude Sonnet 4.6 as judge.
• In a separate evaluation, the instruction-following score rose from 2.0 to 3.4 out of 5.
• This was followed by another round of SFT using rejection-sampled conversations between Claude Opus 4.6 and Talkie.

Future plans include growing the English corpus further, expanding to multiple languages, strengthening leak detection and OCR, and collaborating with historians to build a more sophisticated vintage post-training pipeline. The writeup also lays out a long-term roadmap toward a GPT-3-level model.