Why LLM Research Agents Generalize Without Benchmark Overfitting: 'Compressibility' Is Key

The phenomenon of ML research agents repeatedly optimizing (hill-climbing) benchmarks without overfitting is explained by 'compressibility'. Successful ML strategies can be described by short descriptions that lack the space to memorize training data, indicating that they capture the structural essence of the data.

Generalization Proof via Compression

The research team verified this through the 'Certificate of Output Compression' mechanism. The Explorer optimizes strategies by referring to the validation set, the Compressor compresses these strategies into extremely short prompts (a few tokens), and the Reproducer reproduces the strategies using only these prompts and the training data. If the Reproducer matches the Explorer's performance, it proves that the strategies are compressible, as minimal validation-set-dependent information is transmitted.

Experimental Results and Limitations

• Compression Success Rate: On 8 datasets, most performance was reproduced with 32-token prompts, and LM strategies showed no held-out performance loss even when compressed to 16 tokens.
• Information Bottleneck Verification: The same strategies were discovered even when only 1-bit feedback was provided, and when intentional overfitting was induced, validation-set-specific advantages disappeared after passing through the compression pipeline.
• Conclusion: ML recipes that actually work are simple, and the transfer of performance from fixed benchmarks to new data stems from this compressibility.