LiteFrame Extends Video LLM Efficiency

The biggest challenge in extending Video LLMs to long-form video is the explosive growth of visual tokens. Existing approaches focused on a 'post-hoc reduction' strategy that extracts features and then reduces tokens to lower the LLM load, but in this case the bottleneck shifts to the vision encoder, which performs per-frame processing in the LLM.

LiteFrame is a highly efficient video encoder proposed by Google DeepMind and Seoul National University that fundamentally solves this problem. It uses a new training method called Compressed Token Distillation (CTD) to train a lightweight student model to directly predict the spatiotemporal compressed representations of a large teacher model. It then goes through a Language Model Adaptation (LMA) stage that aligns the compressed latent space with the LLM, enabling processing of up to 512 frames.

Key results:

• 35% reduction in inference latency: Significant reduction in overall inference time combining vision encoding and LLM prefilling
• 71% reduction in parameters: Uses only 87M parameters compared to the teacher model's 304M
• 8x increase in frame throughput: Processes 8x more frames within the same compute budget
• Zero-shot high-resolution handling: Achieves the best score on the HLVid benchmark without high-resolution training

Across multiple benchmarks including Video-MME, MLVU, and LongVideoBench, it presents a new accuracy-latency Pareto frontier, achieving higher accuracy while lowering latency compared to existing models.