Maximizing the Potential of Vision Language Models in Satellite Imagery Through Fine-tuning

Fine-tuning, which adapts foundation models to specific domains, is a key technique for dramatically boosting performance on specialized tasks. In particular, by optimizing a vision language model (VLM) such as Pixtral-12B for satellite imagery data, the limitations of general-purpose models can be overcome.

Instead of retraining the entire model, using LoRA (Low-Rank Adaptation) technology enables efficient training. LoRA injects small trainable matrices into the model weights, effectively teaching domain-specific vocabulary or knowledge without modifying the entire model.

Satellite imagery is a field requiring a high degree of expertise, used in environmental change monitoring, defense, agriculture, and more. A case study using the Aerial Image Dataset (AID) showed that the base model without fine-tuning exhibited the following limitations:

• Difficulty distinguishing fine-grained visual differences, such as between dense residential and medium residential areas
• Hallucination, where the prompt-based approach generated class names that do not actually exist

In contrast, when Pixtral-12B is fine-tuned on satellite imagery data, the model gains an understanding of domain context, enabling much more accurate and fine-grained classification even for ambiguous classes.