VLM Technology Trends and Latest Model Developments

Over the past year, the Vision Language Model (VLM) field has undergone rapid advances, including model miniaturization, enhanced reasoning capabilities, and evolution into multimodal agents.

Major Model Trends

• Any-to-any models: Models that freely input and output various modalities such as text, image, and audio (Qwen 2.5 Omni, MiniCPM-o 2.6) have emerged.
• Miniaturization and efficiency: Smol models that reduce size while maintaining performance, and decoder models adopting Mixture-of-Experts (MoE) architecture, are drawing attention.
• VLA (Vision Language Action) models: These are expanding into models that perform direct actions based on visual information.

Specialized Features and Applications

• Multimodal RAG: Technology for simultaneously retrieving and reranking (Reranking) images and text has advanced.
• Object Detection and Segmentation: Sophisticated object detection and segmentation capabilities using VLMs have been strengthened.
• Multimodal Agents and Video Understanding: Beyond simple image understanding, agent technology that performs video analysis and complex tasks has emerged as a key challenge.

Benchmarks and Alignment

• New benchmarks such as MMT-Bench and MMMU-Pro have emerged, enabling more precise evaluation of model performance.
• New alignment technologies for multimodal safety continue to be researched.