7 Voice Agent Bugs That Script Tests Miss

By operating a production business voice agent built on the Realtime API and analyzing actual call logs (raw transcripts), we identified and resolved 7 behavioral bugs that are difficult to detect with script-based tests.

Key Bugs Found and Solutions

• System Prompt Leakage: When users read out internal instructions, the agent acknowledged and repeated them. We modified the agent to respond only to the user's intent for similar inputs.
• Ignoring Termination Signals: The agent failed to end calls even when users expressed intent to terminate four times consecutively with phrases like "I'm done". We changed the logic to treat non-acceptance responses as refusals and terminate immediately.
• Time Error: The model recalculated injected time values, resulting in a discrepancy of more than 9 hours from the actual time. We configured the model to read injected times as-is and prohibited self-calculation.
• Non-deterministic Extraction: The temperature for structured extraction calls was set to 0.4, causing inconsistencies in action items within the same conversation. We lowered the temperature to 0 for all extraction/classification calls to ensure deterministic behavior.
• Missing Compound Requests: The LangGraph router processed only one response node per turn, causing some responses to be missed. We resolved this by implementing fan-out to multiple nodes, parallel write merging, and message tag-based ordering.
• Cross-topic Memory Loss: The agent failed to recognize facts related to other topics mentioned while answering a current topic, leading to repeated questions. We improved this by extending server-side tracking tools to track coverage per topic and manage durable state.
• Ignoring Fatigue Signals: Even when users expressed dissatisfaction with time or number of questions, the agent politely acknowledged it but maintained the same pacing. We added a switch that stops question-by-question pacing and switches to a mode collecting all information when fatigue signals are detected.

Operational Implications

Reviewing raw conversation logs for one day proved more effective for finding actual bugs than weeks of script-based testing. We recommend regularly reviewing raw logs instead of relying on summaries or evaluation scores when operating production voice agents.