Analysis of Knowledge Cutoffs and Pre-training Timelines for Claude/GPT

By posing sophisticated questions to frontier models and analyzing the responses, it is possible to estimate undisclosed training processes and data compositions. Researchers use this to indirectly determine model parameter sizes, training data mixtures, and pre-training timelines.

Key methods include:

• Incompressible Knowledge Probes: Evaluating accuracy on niche knowledge problems to estimate the parameter scale of models like GPT-5 or Opus.
• Data Mixture Inference: Analyzing tokenization schemes to infer the composition of training data or the data mixtures used in the tokenizer.
• Date and self-identification questions: Measuring up to which point the model has learned information to estimate the training timeline.

Large-scale LLM training typically proceeds in three stages. First, a base model is pre-trained using vast general-purpose data such as the internet. Subsequently, long-form comprehension and specific capabilities are enhanced using textbook-level specialized data. Finally, post-training transforms the model into an assistant persona, refining its personality, reasoning abilities, and tool calling.

The most costly and data-intensive stage is pre-training, which takes several months to create a base checkpoint. During this process, capability and post-training teams conduct experiments to improve the latest checkpoints. Depending on the results, major versions like GPT-4 to GPT-5, or model families like Fable, Opus, Sonnet, and Haiku, may be produced.

Actual released models are the result of combining the latest checkpoint with optimal capability and post-training. However, since publicly available verification materials are limited, these analysis results are all estimates, and some interpretations may differ from the actual training process.