Hardware Aware Dynamic Speculative Decoding

Speculative Decoding(SD) is a technique that speeds up LLM inference by having a large target model verify, all at once, multiple tokens generated by a small draft model. However, it has a limitation: when the batch size (BS) grows and the environment becomes compute-bound, SD can actually end up slower than normal inference.

To address this, Hardware-aware Dynamic Speculative Decoding(DSD) has been proposed, which dynamically determines the optimal number of draft tokens (K) according to hardware constraints. It increases K when memory bandwidth is the bottleneck, and decreases K when compute capacity is the bottleneck, in order to optimize efficiency.

The optimal K value differs depending on the model architecture.

• Dense models: As batch size increases, the optimal K decreases monotonically.
• MoE(Mixture-of-Experts) models: K changes non-monotonically with batch size. At low BS, K is low due to expert loading costs; at medium BS, once experts are already loaded, K increases; and at high BS, it decreases again.

This kind of dynamic adjustment plays a key role in improving overall system throughput, especially during the resource-intensive rollout phase of Reinforcement Learning(RL), and in production environments where request lengths within a batch vary widely.