LLM Mathematical Excellence Driven by Pre-training Data Quality, Not Verifiability

Steven Byrnes argues that the primary reason LLMs excel in mathematics is not 'Verifiability,' but rather the quality of pre-training data based on the high accuracy of mathematical literature itself (where the probability of a random sentence being true is over 99%). He points out that mathematics can establish the foundations of logical reasoning through imitation learning alone, whereas other fields suffer from numerous errors in literature that can confuse learning.

Byrnes emphasizes that RLVR (Reinforcement Learning with Verifiable Rewards) does not generate new mathematical concepts but enables the use of 'legible tools' already secured through pre-training. He analyzes that RLVR cannot elevate performance to IMO level using only elementary school-level data, and that the background behind the leap in mathematical performance lies in the filtering of advanced mathematical data by hired PhD experts and the combination of SFT/RLAIF. He also explains that natural language proofs were already excellent before Lean automation, and that RLVR primarily serves to refine style or meta-cognitive strategies.