talkie releases a language model from the 1930s

talkie is a 13B language model trained solely on pre-1931 English. Two versions have been uploaded to Hugging Face: talkie-1930-13b-base (53.1GB) and talkie-1930-13b-it (26.6GB), with the base model trained on 260B tokens of historical text. Since the training data predates January 1, 1931 under U.S. copyright standards, all of it is in the public domain.

talkie-1930-13b-it is a fine-tuned version created by extracting instruction-response pairs in a simple chat format from classic reference materials with formal structure, such as etiquette guides, letter-writing manuals, cookbooks, dictionaries, encyclopedias, and collections of poems and fables. To improve instruction-following, online direct preference optimization (DPO) was then applied to generation-prompt-based rollouts using Claude Sonnet 4.6 as the judge, followed by one more round of SFT using rejection-sampled multi-turn synthetic dialogues between Claude Opus 4.6 and talkie.

The author calls models built this way—using only data with clear licensing or expired copyright—vegan models. While the base model comes close to meeting this standard, the chat model cannot be considered fully pure due to the subsequent training that leverages modern models.

The core questions this project raises are:

• Can measuring the surprise of explanations of past events reveal performance on future prediction
• Can a model that only knows up to 1911 independently arrive at ideas like general relativity
• Can it write new Python programs, like in HumanEval, having seen only a few examples

The authors are wary of contamination and the anachronistic knowledge injected by modern LLMs. In the long run, they aim to use vintage base models like talkie themselves as judges, building era-appropriate post-training pipelines without relying on modern models.

In a demo, when asked to Generate an SVG of a pelican riding a bicycle, the model responded that the SVG of a pelican riding a bicycle was created in 1860.