AI BriefingPFlash, 10x prefill
Key point
PFlash cut 128K prefill time by 10x compared to llama.cpp on an RTX 3090.
Details
PFlash applies speculative prefill to reduce the long-context prefill bottleneck of quantized 27B-class models. A small drafter scores the token importance of the entire prompt, and the large target model prefills only the important spans.
In a single-shot measurement of Qwen3.6-27B Q4_K_M on an RTX 3090, the TTFT for a 128K prompt was 24.8 seconds, 10.4x faster than vanilla llama.cpp's about 257 seconds. At 64K as well, it improved by 10.0x, from 13.5 seconds versus 134.95 seconds, and NIAH retrieval was maintained throughout.
- vanilla llama.cpp took 248.4 seconds cold and 169.3 seconds warmed for a 131K prompt.
- The implementation is C++/CUDA only, with no Python/Triton/PyTorch used in the inference loop.
- The underlying ideas are Speculative Prefill, Cross-Family Speculative Prefill, FlashPrefill, Block-Sparse-Attention, and ggml/llama.cpp.
- The code was released under the MIT license.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Underdog Releases 'Husky', a Model-Specific Inference Engine Up to 4.5x Faster than MLXNVIDIA Open-Sources Local AI Inference Router 'PAIR'