Phi-4-reasoning-vision and Lessons from Training a Multimodal Reasoning Model

Phi-4-reasoning-vision-15B is a 15B open-weight multimodal reasoning model that spans image understanding, document reading, GUI grounding, and math/science reasoning. It is available on Microsoft Foundry, HuggingFace, and GitHub, and aims for an efficient balance of speed and accuracy rather than being a slow, token-hungry model.

The core direction was not a bigger model but a small, fast VLM. Microsoft explains that, building on lessons learned from Phi-4 and Phi-4-Reasoning, it created a model that handles a wide range of vision-language tasks without excessive data or an oversized architecture. Training used 200B multimodal tokens, far less than Qwen 2.5 VL/3 VL, Kimi-VL, and the Gemma3 family, which used 1T+ tokens.

For architecture, the team chose mid-fusion. Rather than early-fusion, which mixes images and text from the start within one giant transformer, they judged that projecting a pretrained vision encoder into the LLM's embedding space is more practical in terms of compute, memory, and data. In actual experiments, they combined a SigLIP-2-based vision encoder with the Phi-4-Reasoning backbone, and confirmed that extracting information well from high-resolution inputs is important.

To compare vision encoding approaches, they trained a 5B proxy model on 10 million image-text pairs and tested several methods.

• Dynamic resolution was the strongest overall, and especially excelled on high-resolution data.
• The 3600 max tokens setting handles resolutions close to native HD at roughly 720p level, showing large gains on high-resolution benchmarks such as ScreenSpot-Pro.
• Multi-crop with S2 produced better results than regular multi-crop while using fewer visual tokens.
• In the end, the SigLIP-2 Naflex variant was selected.

For data, the focus was on quality over quantity. The final dataset was built around three main axes: refined and improved open-source data, high-quality in-house domain data, and targeted acquisition data. Each dataset's quality was classified by directly reviewing samples, and where answers were incorrect or captions were poor, responses were regenerated using GPT-4o and o4-mini.

Low-quality questions were hard to salvage, but if the image itself was fine, it was reused as a seed for new captions or VQA data. They also boosted data utility by attaching detailed image descriptions to math/science data and mixing instruction-following requirements directly into domain data. Format errors and logical errors were also extensively corrected, so that a small model could learn well even from less data.