Audible uses controllable TTS and BookLLM to improve audiobook character differentiation

Details

Audible's technology team developed a controllable text-to-speech system that uses natural-language descriptions to control dialogue delivery, addressing the inability of conventional TTS to distinguish characters in long-form content.
The system improved dialogue differentiation by 3.7 points on a MUSHRA test and showed statistically significant improvements in listening experience for eight out of ten voices in both English and Spanish.
A new component called BookLLM automatically profiles characters and identifies their dialogue with 97% accuracy in English and 94% in Spanish, using book context to define voice characteristics and assign stage directions.
The approach relies on a novel data synthesis pipeline that generates high-quality speech captions using LLMs and expert models, allowing the TTS model to understand and follow expressive instructions like pitch, rate, and emotion.
BookLLM assigns each character a distinct profile from a curated voice catalogue to maximize contrast between prominent characters, while minor characters fall back to a default narrator to avoid voice proliferation.
The pipeline ensures consistency by fixing a character's voice profile at the book level, preventing the drift seen in general-purpose models where a character's voice changes between chapters.

Key point