Hume AI Unveils Benchmark for Measuring Voice AI Quality

While existing voice AI benchmarks have focused on Latency or Word Error Rate (WER), Real World VoiceEQ places its emphasis on measuring the human-like quality felt in real conversations.

This benchmark evaluates 40+ leading voice models across 15+ dimensions and 60+ metrics, including ASR, TTS, S2S, and voice understanding. In particular, it was built on more than 1 million human evaluation data points to enhance reliability.

Key findings:

• Specialization divergence: There is no single best model; each model shows different specialized strengths, such as technical accuracy, emotional expressiveness, or conversational intelligence.
• Limits of listening ability: Many voice models lack listening ability compared to their speaking ability. While they rely on textual information, they struggle to grasp paralinguistic cues—such as tone, pace, hesitation, and emphasis in the voice—to read the speaker's intent.