How Dash's Relevance Judge Was Optimized with DSPy

Dropbox Dash's relevance judge scores query-document relevance on a scale of 1 to 5, and is used for ranking, training data generation, and offline evaluation. However, quality would fluctuate whenever the model was changed or the prompt was slightly modified, and cost and reliability issues grew together.

Previously, there was a strong judge based on OpenAI o3, but it was too expensive for large-scale processing. So they moved to the cheaper gpt-oss-120b, and used DSPy and the GEPA optimizer to systematically re-optimize the prompt against the difference from human ground truth (NMSE).

The core approach is to fix the task definition, then iteratively incorporate structured feedback by comparing human evaluations with model outputs. In this process, guardrails were also added to prevent overfitting that simply copies the keywords or phrasing of specific examples.

As a result, NMSE decreased from 8.83 → 4.86, a 45% improvement, and the time that used to be spent on manual tuning was reduced from 1-2 weeks → 1-2 days. With the same cost, 10-100x more data could be labeled, which also increased the scale of experiments and statistical confidence.

Operational reliability also improved. In experiments targeting the smaller gemma-3-12b, over 40% of responses were malformed JSON at baseline, but after DSPy optimization, malformed outputs decreased by over 97%, and NMSE also improved from 46.88 → 17.26.

For production judges like o3, instead of a full rewrite, incremental improvement was applied by selecting only safe bullet rules from a validated instruction library and adding them. This made it possible to separately handle the two goals of expanding to alternative models and maintaining the stability of the operating high-quality judge.