LLM Probabilistic Belief Updates Diverge from Bayesian Principles, with Non-Bayesian Heuristics Performing Better

Modern AI systems are being deployed in complex domains such as healthcare, science, and law, where there is no single correct answer, requiring them to rationally update uncertain beliefs as new evidence arrives. Researchers treated LLMs as information processing rules and quantified probabilistic belief inconsistencies within LLMs by measuring the 'information processing gap,' defined as the deviation from Bayes updates.

Discrepancies Between Bayesian Principles and LLMs

Experimental results showed that among the various ways LLMs incorporate evidence into beliefs, some perform nearly perfect Bayesian updates, optimally processing evidence, while others rely on learned heuristics. Notably, non-Bayesian heuristic updates often yielded better results in terms of downstream task performance than accurate Bayesian updates (optimal information processing).

This phenomenon suggests that LLM world models are misspecified. In other words, LLMs often achieve higher accuracy when using their own learned heuristics rather than the Bayesian approach, which is theoretically known to be optimal. The researchers demonstrated that these metrics can be usefully applied to diagnose issues in LLM-based reasoning systems.