Building a Vintage LLM from Scratch (50-minute read)

Using only historical texts from before 1900 as training data, a Vintage LLM with 340M (0.3B) parameters based on the Llama architecture was created. This model aims to recreate the atmosphere of the Victorian era by setting its Knowledge Cutoff to 1900.

The entire pipeline, from data processing to base training and fine-tuning scripts, was built independently. Data processing was performed on a personal PC, while training the 340M model utilized cloud GPUs such as RunPod, ThunderCompute, and Vast.ai. Total GPU costs came to approximately $80.

This model did not undergo any separate alignment or censorship process in order to preserve historical accuracy. As a result, it may contain expressions that could be considered harmful or inappropriate by modern standards, which is a deliberate choice to maintain historical context.