Arbitrage: Efficient Inference via Speculative Decoding Considering Model Superiority

ARBITRAGE is a step-level speculative generation framework that dynamically selects inference paths by determining the relative superiority of a fast but inaccurate draft model and a high-performance target model. Existing speculative decoding suffered from rejecting semantically equivalent reasoning steps due to token mismatches.

Existing step-level approaches also wasted target model computation by repeatedly generating rejected steps. Instead of a fixed acceptance threshold, ARBITRAGE uses a lightweight router to predict the likelihood that the target model will generate a meaningfully better reasoning step.

This router approximates an ideal ARBITRAGE Oracle that always selects higher-quality steps. As a result, it consistently outperforms existing step-level speculative decoding techniques across multiple mathematical reasoning benchmarks, reducing inference latency by up to approximately 2x at the same accuracy.