RidgeFill Achieves 1.25-1.31x Faster Prefill on Qwen3.8-27B Without Training

• RidgeFill is a training-free speculative prefill method that skips computation for upper model layers during the prompt phase.
• On Qwen3.8-27B running on a single A100, time to first token is 1.25 to 1.31x faster for prompts between 16K and 120K tokens.
• The method saves approximately 28 seconds on a 120K-token prompt while maintaining RULER and LongBench v2 scores within 1.2 to 1.4 percentage points of the baseline.
• On Qwen3.8-Flash-Next in vLLM, the method achieves 1.50 to 1.60x speedup at 9K to 32K tokens, with RULER needle tasks scoring 99.8 compared to 100.0 for exact inference.

Methodology

• Uses a per-layer map fitted via ridge regression from a single pass over calibration text to predict late-layer inputs.
• Lower layers run exactly up to a split point, while late layers write their cache from the predicted inputs.
• The full model runs for a chosen set of rows, the last few thousand tokens, and every decoding step.

Availability

• Code is available in a llama.cpp fork and on GitHub.
• Projector weights for Qwen3.8-27B are hosted on Hugging Face.
• Enthusiasts can build custom maps for local inference-grade hardware in 0.5 to 1.5 hours.