What is Speaker Diarization

Speaker Diarization is the process of taking an audio stream with an unknown number of speakers and generating a labeled timeline indicating who spoke when. The system does not know the actual identities of the speakers, but it distinguishes between different individuals and maintains consistent labels throughout the recording.

This technology is central to multi-speaker audio applications where speaker separation is essential, such as meeting minutes, call center analytics, interview transcription, and podcast editing. Diarization is distinct from Speaker Segmentation, which identifies the points where speakers change, and Speaker Identification, which matches voices to actual names.

Most diarization systems follow a four-stage pipeline.

• Voice Activity Detection (VAD): Removes silence, background noise, music, etc., and extracts only actual speech.
• Segmentation: Splits the audio at points estimated to be speaker changes (e.g., changes in timbre, pauses).
• Embedding Extraction: Converts the acoustic features of each speech segment into numerical vectors to create speaker embeddings.
• Clustering: Groups embeddings from the same speaker and assigns labels. Since the number of speakers is unknown beforehand, the algorithm infers it.

The final output consists of anonymous speaker labels linked to time ranges and a transcript. Performance is evaluated using the Diarization Error Rate (DER), which measures overall accuracy, and the Jaccard Error Rate (JER), which checks if accuracy is maintained across all speakers.