Google DeepMind Introduces World's First Double-Blind AI Evaluation to Prevent Benchmark Contamination

Google DeepMind is piloting the world's first AI model evaluation applying a double-blind approach. This measure aims to resolve benchmark contamination issues by ensuring that external evaluators cannot view model weights and that Google cannot pre-view the evaluation data.

This evaluation targets the Gemini Flash Lite model and is conducted through partnerships with the Singapore AI Safety Institute, OpenMains, AVERI, and MLCommons. The core technology used is Confidential Space from Google Cloud's Confidential Computing portfolio, which securely processes both evaluation data and model weights in an encrypted environment.

While existing external evaluations involved evaluators directly inspecting the model or model providers viewing the data, this double-blind method restricts information access for both parties, thereby enhancing the fairness and reliability of the evaluation. This prevents artificial score inflation that may occur during the verification of AI model performance and safety, establishing a foundation for policymakers and enterprises to trust AI benchmark results.