vLLM Publishes Performance Benchmarks for 5 Speculative Decoding Methods on AMD GPUs

vLLM has released a technical guide and benchmark data comparing five major Speculative Decoding methods (Native MTP, Gemma 4 MTP, EAGLE-3, DFlash, DSpark) in environments with AMD Instinct MI300X and MI355X GPUs and the ROCm platform. This document covers the working principles of each method, vLLM configuration methods, and detailed Throughput and Speedup metrics in actual serving environments.

Key Methods and Working Principles

• Native MTP: Uses auxiliary prediction paths embedded in the target model architecture, generating candidate tokens sequentially without a separate checkpoint.
• Gemma 4 MTP: Uses a separate checkpoint paired with the target model, sharing the target model's Activation and KV Cache.
• EAGLE-3: Uses a dedicated network that performs Drafting in an Autoregressive manner based on the target model's Hidden States.
• DFlash: Predicts candidate tokens in parallel block units conditioned on the target model's Hidden States to reduce Drafting latency.
• DSpark: Adds a lightweight Sequential Head and Confidence-based Prefix selection mechanism to DFlash's parallel backbone to improve consistency.

Performance Benchmark Results

Experiments were conducted on various model families including Gemma, Qwen, Kimi, and MiniMax, targeting GSM8K, MATH500, HumanEval, and MBPP benchmarks.

• Maximum Speedup: Some settings achieved more than 2x throughput improvement compared to the Non-speculative Baseline. For example, on the gemma-4-26B-A4B-it model, the DFlash method recorded a 2.87x speedup on the MATH500 benchmark, and Gemma 4 MTP recorded a 2.74x speedup on GSM8K.
• Model-Specific Characteristics: Native MTP on the Qwen3.5-122B-A10B model showed a 2.20x speedup on MATH500, while DFlash on the Kimi-K2.5 model showed a 2.68x performance improvement based on MATH500.
• Limitations of Smaller Models: On smaller models like Qwen3-8B, Drafting overhead led to performance degradation or only marginal improvements compared to the Baseline on some benchmarks (such as MATH500).

Tuning and Deployment Considerations

Speculative Decoding is a target for runtime optimization rather than a fixed configuration. The optimal value for num_speculative_tokens (proposal length) varies by model and workload, and excessive values can increase Drafting costs, thereby reducing throughput.

• Recommended Tuning Workflow: Start with the checkpoint's recommended settings, run benchmarks with representative prompts, and monitor Throughput, Mean Accepted Length, and Acceptance Rate to find the optimal num_speculative_tokens.
• Monitoring Metrics: If the Acceptance Rate for later Draft Positions is low, it is advantageous to reduce the proposal length to avoid unnecessary Drafting work.