DeepMind releases ProEval GitHub repository for GenAI evaluation

ProEval is a framework that targets both performance estimation and failure case detection in GenAI evaluation. It is designed to estimate model error rates using only a limited number of samples, while also more aggressively surfacing diverse failure patterns. By leveraging a pretrained GP surrogate, it can be applied immediately to new models, and it aims for an accuracy level of about ±1%.

The key point is not simply producing scores faster, but revealing more bugs and vulnerabilities even under small-budget settings. It explains that this can be easily attached to GenAI evaluation systems across domains such as reasoning, safety, and classification, as well as across various modalities.

In the example code, a sampler is created with BQPriorSampler(noise_variance=0.3), and then sample(predictions="svamp", target_model="gemini25_flash", budget=50) is called to obtain an estimate. Afterward, load_predictions() and extract_model_predictions() are used to load the actual prediction matrix and compare it against the real error rate.

Experiments can be run with python -m experiment.exp_performance_estimation --dataset svamp --n-runs 5, and it states that this has been validated on GSM8K, MMLU, StrategyQA, Jigsaw, and others. The repository is distributed under Apache 2.0 for software and CC-BY for materials, and it explicitly states that this is not an official Google product.