AI BriefingNeuroFlow: 55x Faster ViT Video Inference
Key point
NeuroFlow is proposed to maximize the inference speed and efficiency of Vision Transformers by using EMA to remove redundant tokens in video.
Details
NeuroFlow is a dynamic routing framework that leverages the temporal redundancy present in video inference to improve the computational efficiency of Vision Transformer (ViT). It tracks semantic surprise through the Exponential Moving Average (EMA) of patch-level embeddings, and physically removes static tokens such as background before the encoder stage.
Key results are as follows:
- Architecture C (Dual-Memory Reconstruction): A training-free inference engine that requires no separate training. On the SigLIP model, it achieved 84.0% token sparsity while maintaining 92.4% of the original dense model's accuracy, recording a zero-shot top-1 accuracy of 71.55%.
- Architecture B (Extreme Speedup): Provides dramatic speed improvements by removing static tokens in advance. For 1792p SigLIP 2 inference, it achieved about a 55.80x wall-clock execution time reduction, from 678ms to 11.9ms.
- LLM Scalability: When applying a similarity gate bypass to language models such as Phi-3-mini, the technical boundary was validated by confirming 0% token drift during grammar-constrained generation.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
MultiMatte, a Promptable Background Removal Model Based on SAM 3, Released, Achieving S-measure of 0.901 on DIS-VD Benchmark