Improving Quality and Robustness of LLM-based Text-to-Speech Systems

LLM-based TTS produces natural-sounding speech from just short voice samples, but accent leakage, lack of expressiveness, and reliability issues still remain. To address this, Amazon combined locale-specific data augmentation, LoRA fine-tuning, classifier-free guidance (CFG), and chain-of-thought based verification.

The biggest challenge in polyglot TTS is accent leakage. When transferring an English voice to French, German, or Spanish, the original speaker's accent bleeds through, or conversely, the target language's accent overrides the speaker's own vocal identity. Amazon applied LoRA with data that significantly increased the proportion of target locale content, achieving accent-free voice cloning that produces natural pronunciation in the target language while preserving speaker identity.

Expressiveness was enhanced with CFG. The system generates synthetic reference audio to better surface elements like emotion, laughter, sighs, and hesitation, and uses this as an inference condition to induce richer prosody. Since this approach trains speaker identity and accent separately, it can scale to more locales and languages with only a small number of recorded voices.

Results were presented as well. Based on MUSHRA listening evaluations, quality improved by 5%~20% across 9 locales compared to the previous model family.

• US-English: +12.43%
• Southern US-English: +20.05%
• Great Britain-English: +5.97%
• Australia-English: +5.50%
• US-Spanish: +11.78%
• Spain-Spanish: +13.23%
• France-French: +8.44%
• Germany-German: +14.12%
• Italy-Italian: +9.80%

The reliability issue is trickier. Traditional TTS has explicit stages like grapheme-to-phoneme, duration prediction, and acoustic generation, but autoregressive LLM-based TTS generates speech tokens one at a time without length planning. This results in hallucination, random truncation, and pronunciation inconsistencies.

To address this, Amazon inserts chain-of-thought reasoning before generation, having the model first predict the phoneme sequence and duration. This prediction is useful in two ways:

• Phoneme prediction: Handles heteronyms like "read", "lead" and unfamiliar names more reliably
• Duration prediction: Plans overall length and per-phoneme timing, reducing hallucination and truncation

Guardrails are added on top of this, checking after generation whether the output length and duration match the prior predictions. If the output length is abnormally long or short relative to the phoneme count, it's flagged as an issue, and if necessary, the system regenerates with different sampling parameters or falls back to an alternative path.

Data curation was also carried out in parallel. Using simple ASR-based filtering alone risks discarding not only well-matched data but also some expressive samples, so Amazon used ASR metrics together with attention mechanism-based metrics to preserve well-aligned data while retaining expressiveness.

With this combination applied, critical errors in typical long-form text were reduced to an average of less than 1 second per hour. Here, critical errors refer to hallucination, cutoffs longer than a single word, and mismatches between input text and output speech. Ultimately, the conclusion is that while LLM-based TTS sounds more natural than traditional systems, deploying it requires additional design work to reinforce accent, expressiveness, and stability all together.