Flash Vision-Language-Action Inference for Autonomous Driving: FlashDrive

Traditional autonomous driving separates perception and planning, but this approach easily breaks down in rare and complex long-tail situations. Vision-Language-Action (VLA) models solve this problem with an end-to-end approach that includes chain-of-thought reasoning, and NVIDIA recently released open-source reasoning VLAs, Alpamayo 1 and Alpamayo 1.5.

The problem is speed. Alpamayo 1.5 (10B, based on Qwen3-VL) takes 716ms per step on an NVIDIA RTX PRO 6000, falling well short of real-time driving requirements. To address this, FlashDrive uses algorithm-system co-design that touches all four stages—encode, prefill, decode, and action—reducing overall latency to 159ms, corresponding to a 4.5× speedup.

The bottleneck wasn't in a single place. VLA inference is divided into vision encoding, prompt prefilling, reasoning token decoding, and flow matching-based action generation, and profiling revealed that all four stages were meaningfully slow. In particular, decode and action accounted for a large share of the overall time, but encode and prefill were also substantial, meaning a single optimization couldn't achieve real-time performance.

The first thing applied was streaming inference. Since autonomous driving input consists of continuous video streams from multiple cameras, re-encoding the entire window at every step is wasteful. FlashDrive reduced redundant computation as follows:

• KV cache reuse to eliminate redundant computation across the previous 3 frames
• Pre-RoPE key caching aligned with frame movement, combined with on-the-fly rotary embeddings
• Design of a custom streaming attention mask to handle camera view-major token ordering

This approach reduces effective sequence length by 75%, but since the streaming KV cache isn't exactly identical to a full forward pass, accuracy can suffer. Interestingly, fine-tuning the entire VLM with streaming input made things worse, because the action expert is more sensitive to the full KV cache than reasoning tokens are. So FlashDrive shifted to keeping the VLM fixed and fine-tuning only the action expert, and to reflect the accumulated error that arises in deployment, it recovered performance by filling the cache across multiple streaming steps and propagating gradients only at the final step.

For the reasoning stage, speculative reasoning was applied. Since driving-domain reasoning sequences are short—about 16 tokens—and strongly templated with low entropy, the block diffusion model DFlash is used as a parallel drafter to generate candidate blocks all at once. Thanks to speculative verification, the output distribution remains identical to conventional autoregressive decoding, reducing the decode bottleneck without any quality loss.

Finally, adaptive-step flow matching was applied to action generation. Rather than using the standard 10-step denoising as-is, analysis of the velocity field revealed a U-shaped pattern, where change is large at the start and end and nearly constant in the middle. Based on this, trajectory generation was accelerated by reducing redundant steps in the middle range while preserving the important endpoints more fully.

The key point is that it's not a single optimization, but rather that streaming inference + speculative reasoning + adaptive action generation were combined to reduce the entire pipeline together. As a result, FlashDrive brings reasoning VLAs for autonomous driving much closer to real-time while largely preserving accuracy.