VibeVoice - Open-Source Frontier Voice AI

VibeVoice is a family of voice AI models covering both TTS and ASR, targeting long-form speech generation and long-form speech recognition.

The core architecture combines an ultra-low-frame-rate continuous speech tokenizer at 7.5 Hz with next-token diffusion. An LLM handles textual context and conversational flow, while a diffusion head handles acoustic detail.

• VibeVoice-ASR processes up to 60 minutes of audio in a single pass, producing structured transcripts that include speaker information, timestamps, and spoken content.
• It supports 50+ languages and is designed to accept custom hotwords and contextual information to improve domain-specific recognition accuracy.
• Finetuning code and vLLM inference support have been released, and it can also be used via Hugging Face Transformers.

VibeVoice-TTS synthesizes up to 90 minutes of speech in a single pass and supports up to 4 speakers in a single conversation. However, the repository notice states that after release, misuse inconsistent with its intended purpose was identified, and the TTS code has been removed.

VibeVoice-Realtime-0.5B supports streaming text input, a first-audio latency of about 300ms, and generation of long-form speech about 10 minutes long. It also adds multilingual experimental voices in 9 languages and 11 English-style voices.

Weights are distributed via Hugging Face, Playground, and Colab, and the repository is designated for research and development use only. It may inherit biases and errors from its base model, Qwen2.5 1.5B, and responsible use is recommended due to risks of deepfakes, impersonation, and misinformation.