Microsoft releases VibeVoice

Microsoft has released VibeVoice-ASR. It's a Whisper-family speech recognition model with built-in speaker diarization, distributed under the MIT license.

On a 128GB M5 Max MacBook Pro, using uv and mlx-audio, the MLX-converted mlx-community/VibeVoice-ASR-4bit (5.71GB) was run. The original VibeVoice-ASR model is 17.3GB, and with --max-tokens 32768, the first 1 hour of a 99.8-minute podcast was transcribed into JSON segments.

Performance was as follows.

• Processing 1 hour of audio took 524.79 seconds.
• Peak memory reported by the tool output was 30.44GB.
• Activity Monitor showed 61.5GB during the prefill stage and about 18GB during the generation stage.

The constraints were also clear.

• The default --max-tokens value is 8192, which only covers about 25 minutes.
• Both .wav and .mp3 worked fine, but VibeVoice itself only handles up to 1 hour.
• Longer files require splitting with overlap and aligning speaker IDs.

The resulting JSON is an array including speaker_id, making it easy to explore in Datasette Lite. Interestingly, three speakers were detected — besides the two people in the conversation, a separate Lenny voice used for sponsor reads was distinctly identified.