AI BriefingNeuroFlow: 55x Speedup for Video ViT
Key point
By leveraging EMA to eliminate redundant tokens in video, it dramatically boosts the inference speed of Vision Transformers.
Details
NeuroFlow is a dynamic routing framework that exploits the temporal redundancy of video data. It tracks 'semantic surprise' via the Exponential Moving Average (EMA) of patch-level embeddings, and physically removes static tokens—such as background—before the encoder.
Key Features and Results:
- Dramatic speedup: In high-resolution 1792p SigLIP 2 inference, it achieved roughly a 55.8x real-time speedup, going from 678ms to 11.9ms (while maintaining 97.37% embedding fidelity).
- Training-free: Without any weight modification, it uses a dual-memory reconstruction structure combining a 'Layer 0 Retinal Gate' and a 'Layer 12 Cortical Cache'.
- High efficiency: Even at 84.0% token sparsity, it achieves 71.55% zero-shot top-1 accuracy on SigLIP, retaining 92.4% of the dense model's accuracy.
It also demonstrated scalability across models by showing that applying a similar gating approach to LLMs such as Phi-3-mini results in almost no token drift.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.