AWS and Booking.com combine knowledge distillation and Medusa to cut LLM costs by two-thirds while speeding up inference up to 20x

Details

Combining knowledge distillation (KD) and the Medusa framework allows a compact LLM to match the quality of a much larger teacher model while running 10 to 20 times faster and at less than one-third the cost.
The study used TinyLlama (1.1 billion parameters) as the student and Llama-3.1 (70 billion parameters) as the teacher, applying the techniques to Booking.com's AI Trip Planner, Smart Filters, and Accommodation Description Generation features.
Knowledge distillation closed the performance gap between the student and teacher models by more than 90%, while adding Medusa heads provided an additional 2 to 3.6 times speedup in inference.
In production, the optimized models achieved a 2.9% increase in user engagement for the AI Trip Planner, 15 times faster responses for 99% of Smart Filters queries, and 1.4% more helpful ratings for accommodation descriptions at less than one-third the cost of GPT-4.
The work was presented at the Association for Computational Linguistics (ACL) and received the Industry Track best-paper award.
Medusa accelerates inference by predicting multiple tokens in parallel using added prediction heads, while KD transfers knowledge from a large teacher model to a smaller student model using labeled and machine-generated data.

Key point