How to Evaluate Voice Agents with LangSmith

Voice agents must converse naturally and resolve user issues while simultaneously achieving their designed business goals. To evaluate them, you need to examine the full trace, recordings, and tool activity, going beyond simple conversation transcripts.

Evaluation criteria are divided into three categories.

• Execution: Verifies whether the agent followed instructions and policies.
• Outcome: Assesses whether the interaction achieved its intended goal.
• Experience: Examines whether the call was natural and smooth.

In execution evaluation, you check whether the correct tools were called with the right arguments and in the correct order, whether privacy, disclosure, and consent policies were adhered to, and whether necessary information was collected and confirmed. The evaluation covers not just the final answer but also the process by which the agent arrived at that answer.

For requirements that can be defined by clear rules, a code evaluator is suitable. For example, you can quickly and cost-effectively check if check_availability was called before book_appointment, if the confirmed date, time, and timezone were passed to the booking tool, and if required disclosures were included in the transcript.

For requirements that require judging the meaning of natural language, use an LLM judge. This can evaluate policy compliance, answer accuracy, whether necessary information was requested, the clarity of explanations for next steps, and appropriate follow-up questions for ambiguous requests.

LLM judges operate more reliably with narrow-scope, clearly defined rubrics—such as "Did the agent confirm the date, time, and timezone before calling the booking tool?"—rather than broad questions like "Was the response good?" In LangSmith, you can score conversations based on complete traces and specific rubrics, and compare quality changes over time as updates are made.