AI BriefingApple Research Proposes Two-Stage Distillation to Convert Transformers to Mamba
·2026.11.11 22:29
Key point
A new recipe distills Pythia-1B into a Mamba model, achieving a perplexity of 14.01 compared to the teacher's 13.86.
Details
- Researchers from Apple, Concordia University, and Johns Hopkins University introduce a two-stage method to distill Attention-based Transformers into Mamba-like State Space Models (SSMs) without using hybrid Attention blocks.
- The approach first distills a Transformer into a linearized Attention version using a kernel trick adaptation, then distills that into an adapted Mamba model.
- The distilled Mamba model retains most of the original Pythia-1B performance, reaching a perplexity of 14.01 versus the teacher's 13.86.
- Performance gaps appear primarily on downstream tasks that rely heavily on in-context recall.
- The study includes ablations at 1B scale with 10B tokens, analyzing sequence mixer architectures, model size scaling, and token allocation between distillation stages.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
AWS introduces priming to convert pre-trained Transformers into hybrid state space models with 99% less pretraining cost
Meta Research Engineer Describes RSI Work and Skills for AI Research Careers