The Problem with 'Mathematically Proven' Claims About LLMs

In AI discourse, papers that contain actual mathematics repeatedly get turned directly into conclusions like 'LLMs can't do X.' This piece examines how this pattern appears with the same structure in three recent papers.

• Hector Zenil's self-improvement paper models KL-divergence-based retraining as a dynamic system in which external signal diminishes as α_t → 0, demonstrating model collapse. However, the paper explicitly states that systems with an external verifier, fixed axioms, or a formally specified environment fall outside this conclusion.
• Xu, Jain, Kankanhalli's 'Hallucination is Inevitable' defines hallucination as failing to exactly reproduce the output of a computable function, concluding that LLMs trained only on input-output pairs will inevitably produce errors when used as general-purpose problem solvers. However, the theorem doesn't apply to systems that receive external information, such as retrieval, tool use, knowledge bases, or formal verifiers.
• Varin Sikka, Vishal Sikka's 'Hallucination Stations' presents a limitation, arguing that tasks requiring more than O(N^3) operations within a length-N prompt cannot be handled by a transformer's forward pass alone. However, the authors themselves acknowledge that composite systems combining multiple LLMs, or combinations with tools, can achieve higher capability.

In practice, the approaches that actually improved performance were loops that fed in external signals, such as AlphaZero's self-play, RLVR, and verifier-filtered synthetic data. The piece therefore emphasizes that instead of headlines claiming 'mathematically proven,' one should read precisely which limitations hold under which assumptions.