Aurora

When operating LLMs, the trade-off between performance and cost is an unavoidable challenge. Speculative Decoding is the standard method for speeding up inference, but it has limitations: the draft model's performance degrades as traffic patterns change, and offline retraining can't keep up with real-time traffic.

Aurora is an open-source RL (Reinforcement Learning) based framework that learns directly from real-time inference traces. It asynchronously updates the Speculator without interrupting serving, transforming speculative decoding from a static configuration into a dynamic, self-improving system.

Aurora offers the following key benefits:

• Directly addresses the distribution mismatch problem, achieving an additional 1.25x speedup compared to existing static baselines
• Reduces infrastructure costs by eliminating large-scale activation data collection pipelines
• An algorithm-agnostic framework design that will be compatible with various future speculators
• Supports heterogeneous user environments with diverse requirements

Experimental results showed higher performance than well-trained static speculators on major models such as Qwen3 and Llama3. In particular, it demonstrated that performing online learning from scratch can outperform elaborately pretrained static baselines.