AI BriefingHow Dropbox Used DSPy to Turn Dash Chat's AI Evaluation into Better Responses
Key point
Dropbox leveraged DSPy to refine its LLM-based evaluation model, thereby improving both the response quality and efficiency of the Dash Chat agent.
Details
Dropbox's AI feature, Dash Chat, is an agent-based service that pulls information from various sources such as documents, messages, and meeting notes to answer user questions. To measure the quality of the agent, Dropbox built an LLM-as-judge system that evaluates not just the final response but the entire process, including intent understanding, tool use, context gathering, and grounding.
Agent evaluation is far more complex than existing search relevance evaluation, which judges a single output. This is because agents go through a multi-step process that includes Tool calling and Multi-turn conversations. Therefore, a sophisticated layer is needed to individually evaluate each component, such as intent understanding, tool use, context selection, synthesis, and grounding.
Dropbox adopted the DSPy framework and used a two-stage strategy to optimize the evaluation process.
- Stage 1: Refining the Judge model: Using a small amount of human-labeled data as a benchmark, Dropbox applied DSPy's optimization algorithms (GEPA, MIPROv2) to calibrate LLM evaluation scores to align with human judgment.
- Stage 2: Agent optimization: Using the improved judge model, Dropbox built a feedback loop to optimize the System Prompt of the Dash Chat agent.
Through this approach, Dropbox achieved results that improved response completeness while reducing Token usage without sacrificing quality.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.