Apple Research Compresses Streaming Neural Audio Encoders via Latent-Space Distillation

Apple researchers developed a method to compress streaming neural audio encoders, which are critical for on-device dictation, by using latent-space distillation. The approach targets the pre-quantizer latent representation—the last shared interface between the tokenizer and the language model—rather than discrete tokens or output distributions.

Distillation Methodology

The technique trains a student encoder to regress the teacher’s per-frame latent using a squared-error objective. A single affine layer absorbs the width mismatch between the teacher and student models. Because the target precedes both the quantizer and the language-model bridge, this single recipe applies to both supported token interfaces and works for tokenizers trained alone or jointly with a language model.

Performance Results

At 2.8× compression, the distilled student model stays within 1.9% relative WER of its teacher on five of six teacher–student pairs without any fine-tuning. Additionally, the distilled model improves upon an independently trained tokenizer of identical capacity by 3.9% relative.