Improving the Readability of Language Model Outputs via Prover-Verifier Games

When language models solve complex problems, optimizing solely for correctness can make the resulting outputs difficult to understand. In fact, when humans evaluated solutions from a model optimized only for correctness, the error rate was about 2x higher than for a model that was not.

To address this, the Prover-Verifier Game framework was introduced. This approach trains through interaction between a Prover (a strong model) that generates solutions and a Verifier (a weak model) that checks their correctness.

The training process consists of an iteration of two stages:

• Verifier Training: The weak model is trained to accurately predict whether solutions generated by the Prover are correct.
• Prover Training: The strong model is guided to generate solutions that are both correct and easily verifiable by the Verifier—in other words, highly readable.

With this method, the gain in model performance may be smaller than with correctness-only optimization, but it achieves Legibility, allowing humans and other AI systems to understand and verify the outputs far more effectively.