How to Boost Dash Search Relevance by Amplifying Human Labeling with LLMs

Dropbox Dash combines search and answer generation using the RAG pattern. First, enterprise search finds documents, and only some of them are passed to an LLM to generate answers. So the quality of the final answer heavily depends on the search ranking, and on the quality of the relevance label used to train that ranking.

Instead of humans writing rules by hand, the ranking model is trained using machine learning methods like XGBoost. Query-document pairs are labeled on a 1~5 point relevance scale, and the model learns from these examples to rank more useful results higher. The key here is securing enough high-quality labels.

Labels can be obtained in three ways.

• User behavior signals: inferred from signals like clicks and skips
• Human labeling: scores assigned directly by people
• LLM evaluation: relevance judgments generated by an LLM

However, user behavior has bias and sparsity, and human labeling is expensive and hard to scale. In particular, for sensitive or company-specific internal documents, it's practically impossible for humans to evaluate them at scale directly. LLMs, on the other hand, are cheaper, more consistent, and can handle larger candidate sets and multiple languages, but they strongly depend on quality and instruction design, so they must be calibrated with human review.

Dash first validates the LLM using a small amount of human-labeled data, then adjusts prompts and parameters based on the results. Once it passes the bar, the LLM generates hundreds of thousands to millions of relevance labels, which are used to train the ranking model. In this structure, humans teach the LLM, and the LLM creates large-scale training data in return, amplifying human effort by up to 100x.

The LLM's quality is evaluated based on its agreement with human labels. When they match exactly, the loss is 0, and in the worst-case mismatch, mean squared error (MSE) rises as high as 16. Small differences incur small penalties and large differences incur large penalties, and this approach is used to iteratively improve prompts and models.

Rather than blindly increasing evaluation data, the focus is on cases likely to reveal errors. For example, priority is given to analyzing cases where a user clicks on a document the LLM rated low, or consistently skips a document the LLM rated high. These mismatches are used to prioritize human review and prompt improvements.

Accurate relevance judgments often require more than just the document and query text. Inside Dropbox, an expression like "diet sprite" might not refer to a beverage but rather an internal performance management tool, and abbreviations can mean different things across organizations. That's why Dash provides tools that let the LLM first investigate the query context, enabling more consistent context-aware labeling.

Prompt optimization is another important axis. There's an ongoing loop of reviewing misclassified cases, revising instructions, and re-evaluating, and meta-prompting frameworks like DSPy help automate this process. The MSE improvement is presented as a combined result of prompt refinement, adoption of a reasoning-optimized model, addition of query context, and DSPy optimization.

The core conclusion is clear. LLMs are not a replacement for human judgment, but a tool for scaling a small set of high-quality human labels into a measurable, auditable, and correctable large-scale evaluation system. Even as Dash expands to more content types like images, video, messages, and chat, human-calibrated LLM evaluation remains the common mechanism for consistently scaling relevance.