Fine-Tuning Vision-Language Models on Memory-Constrained Devices

Fine-tuning vision-language models (VLMs) usually relies on backpropagation, but this incurs heavy computation and memory costs, making it burdensome on resource-constrained edge devices. An alternative, zeroth-order (ZO) methods, use only forward passes, but existing approaches have fallen far short of backpropagation-based training in terms of accuracy and convergence speed.

The core problem lies in ZO's high variance and local search characteristics. Estimated gradients tend to become unstable, and the method can get trapped in local optima without finding better global solutions on the loss landscape.

In response, NeurIPS 2025 proposed SharpZO. SharpZO is a hybrid sharpness-aware zeroth-order optimization that fine-tunes VLMs using only forward passes, and it operates in two stages.

• Stage 1: global exploration

  • Uses CMA-ES (covariance-matrix adaptation evolution strategy) to smooth out sharpness in the loss landscape.
  • Jointly updates the mean and covariance matrix of the distribution, and adds a term reflecting the worst-case loss that could arise from the current distribution to the loss function, creating a smoother starting point.

• Stage 2: local search

  • Performs refined search using a modified sparse ZO.
  • Instead of simply discarding small gradient terms as before, it normalizes the gradient vector based on its mean and standard deviation, reducing outlier estimates and finding the optimum more stably.

In evaluation, 11 downstream tasks were tested using a CLIP backbone. SharpZO showed up to 7% higher average accuracy than forward-only methods such as ZIP and BlackVIP, and on some tasks it approached the performance of CoOP, which requires backpropagation.

Speed also improved. On ImageNet, SharpZO reached the target accuracy in just 15.3 minutes, faster than ZIP's 19 minutes and BlackVIP's 170 minutes. Because it avoids gradient storage, memory usage is also lower, and it showed stronger performance than baselines even under distribution shift settings such as ImageNet-Sketch and ImageNet-A.

However, SharpZO is currently optimized for prompt tuning, and extending it to full-model fine-tuning remains a task for the future. Also, the sharpness-aware CMA-ES warmup stage requires coordinate-wise gradient estimation (CGE), which can become computationally expensive in high-dimensional settings. For now, this method is therefore particularly well-suited for parameter-efficient fine tuning (PEFT).