Routing-Based Approach for Real-Time Multilingual ASR Unveiled

To address the high computational cost of large multilingual models and accuracy issues during language switching, a routing-based approach that combines small single-language models has been proposed.

This system utilizes specialized single-language models with roughly 100 million (100M) parameters, with the following key components:

• Zipformer: performs low-latency streaming transcription
• Silero VAD: detects speech boundaries
• SpeechBrain: performs language identification (LID)

The coordinator buffers audio while monitoring language confidence, and when a language switch is detected, it rolls back to the last speech boundary and re-transcribes with the correct model to ensure accuracy.

Benchmark results showed that for inter-utterance language switching, the system recorded a WER (Word Error Rate) of about 13%, outperforming existing cloud APIs. For intra-utterance language switching, it recorded a WER of about 41%, but still maintained superior performance compared to open-source alternatives.