AI BriefingOn-Device ASR Performance Improvement
Key point
On-device streaming ASR achieved **8.20% WER** and **0.56s** latency on CPU.
Details
Pushed the accuracy, latency, and memory trade-off of on-device streaming ASR forward in a CPU-only environment.
- Compared 50+ configurations across Whisper, Nemotron, Parakeet TDT, Canary, Conformer Transducer, Qwen3-ASR, etc., in batch, chunked, and streaming modes.
- As a result, NVIDIA Nemotron Speech Streaming was identified as the strongest candidate for real-time English streaming.
The pipeline was reimplemented with ONNX Runtime, applying importance-weighted k-quant, mixed precision, round-to-nearest quantization, and graph-level operator fusion.
Model size was reduced from 2.47GB → 0.67GB, while WER stayed within an absolute 1% of full-precision PyTorch.
The recommended configuration, int4 k-quant, achieved an average 8.20% streaming WER and 0.56s algorithmic latency across 8 standard benchmarks, presenting a new quality-efficiency point faster than real-time on CPU.
Source: https://arxiv.org/abs/2604.14493
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Wispr Unveils 'Canto', a Real-Time Transcription-Specialized Voice Model Achieving Lower WER Than Competitors via GRPO Training