AI BriefingA Theoretical Framework for Acoustic Neighbor Embeddings
Key point
It gives a probabilistic interpretation of distances in acoustic neighbor embeddings and validates it on vocabularies of up to 500k words.
Details
A theoretical framework is presented for interpreting Acoustic Neighbor Embeddings as representations that capture the phonetic content of speech and text in a fixed-dimensional space. The distance between embeddings carries a probabilistic meaning grounded in a quantitative definition of phonetic similarity between words.
This framework rests on theoretical and empirical evidence that the embeddings are close to uniform cluster-wise isotropy, which in turn allows the distance to be reduced to a simple Euclidean distance. This preserves the rich interpretation while making practical application much simpler.
4 experiments were conducted for validation.
- Isolated word classification via nearest-neighbor search between audio and text embeddings achieved the same accuracy as an FST even on a vocabulary of 500k scale.
- For out-of-vocabulary word recovery, the accuracy gap compared to phone edit distance was only 0.5% point.
- English dialect clustering produced the same clustering hierarchy as human listening experiments.
- The embedding distance interpretation also proved valid for predicting the expected confusion of wake-up words.
All source code and pretrained models are provided together, so the work can extend beyond theoretical validation into practical use.