Talkie, Trained on Pre-1931 Text

Talkie is a 13B LLM released by Alec Radford, Nick Levine, and David Duvenaud.

The training corpus was restricted solely to text published before December 31, 1930. It excludes the internet, Wikipedia, and modern events, creating a distribution that doesn't rely on modern web documents.

The key experimental setup is as follows.

• Claude Sonnet 4.6: evaluator for the online DPO pipeline
• Claude Opus 4.6: generation of synthetic multi-turn conversations for final fine-tuning
• The team views this use of modern LLMs as a data contamination risk and stated they plan to remove it in subsequent versions.

Even with zero modern code present, the model showed the ability to write Python code using only 19th-century math texts and a few in-context examples, and the research focus is on drawing the line between memorization and generalization. Application questions center on what the model can learn outside of modern web data, such as long-term forecasting, future inventions, and LLM identity.