1930s-Knowledge LLM Released

talkie-1930-13b is a 13B vintage LLM trained on a corpus of 260 billion tokens, not 26 billion, of pre-1931 English text. Compared to a modern twin of the same architecture (trained on FineWeb), it lags behind in knowledge evaluations, but excluding questions anachronistic to the 1930 timeframe, the performance gap narrowed by about half.

• As a contamination-free experimental model, it was used to measure generalization and prediction ability from a past point in time.
• On HumanEval-style Python tasks it was far weaker than modern models, but it increasingly solved simple one-line programs and small edit-type problems well.
• Cases were also found where later knowledge leaked in, such as President Roosevelt and the New Deal, making temporal leakage filtering a key challenge.

Data quality was also a major bottleneck. Historical text transcribed with existing OCR had training efficiency of about 30% compared to human transcriptions, and even after regex cleanup it reached only about 70%. To improve this, a vintage OCR system is being developed.

For post-training, instead of modern chat data, they reconstructed formal Q&A, summaries, and dialogue data from historical documents. Adding synthetic prompts and DPO based on Claude Sonnet 4.6 judgments raised the instruction-following score from 2.0 to 3.4/5.

Going forward, the team aims to scale up to a GPT-3-level model and a historical corpus of over 1 trillion tokens.