How to Make Text-to-Speech Models Respond in Under 50ms

Through the implementation of Qwen3-TTS 1.7B CustomVoice, we achieved 10 RPS throughput and sub-50ms p95 TTFA (Time-To-First-Audio) on a single NVIDIA H100 SXM. This provides cost efficiency of approx. $2/1M chars, compared to ElevenLabs V3 ($100/1M chars) or Cartesia Sonic 3.5 ($49/1M chars).

A comparison of five existing engines, including vLLM-Omni, SGLang-Omni, and VoxServe, revealed that only this implementation maintained sub-50ms p95 TTFA after low-latency streaming tuning. It sustained sub-100ms latency even at 20 RPS, demonstrating a generation performance of approximately 630 characters per second at 10 RPS.

Key optimization techniques include:

• Leading Silence Removal: Dynamically trimming silent intervals in the initial PCM to reduce latency by approximately 80ms
• Frame Accumulation Tuning: Lowering TTFA with small initial chunks and increasing chunk size thereafter to ensure playback continuity
• Integrated Scheduler: Integrating three modules—Talker, Code Predictor, and Causal Codec—into a single scheduler to optimize coordination between heterogeneous tasks

The implementation and benchmark code have been released as open source.