AI BriefingSegmented Attention Decoding for Long-Form Audio Encoding
Key point
The paper proposes four improvements to address the loss of positional information that occurs when AED models process long audio.
Details
Existing Attention-based Encoder-Decoder (AED) models, when trained on speech data segmented into short units, implicitly learn absolute frame positions by using context beyond the segment boundaries. However, when decoding actual long-form audio, this contextual information disappears, causing the model to lose track of the speech encoding order during cross-attention due to the Permutation Invariance property of keys and values.
To address this problem, this study proposes four key modifications:
- Injecting Explicit Absolute Positional Encodings into the cross-attention of each decoding segment
- Performing Long-form Training using extended audio context to remove implicit positional encoding
- Applying Segment Concatenation to handle various segmentation schemes
- Introducing Semantic Segmentation to align AED decoding segments with training segments
Through these improvements, the accuracy gap between continuous audio encoding and segmented audio encoding was reduced, enabling the use of auto-regressive decoding with attention decoders.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Wispr Unveils 'Canto', a Real-Time Transcription-Specialized Voice Model Achieving Lower WER Than Competitors via GRPO Training