VLM Optimization and Execution Guide for Intel CPUs

This covers the optimization process for efficiently running a Vision Language Model (VLM) on an Intel CPU in a local environment. The core idea is to use Optimum Intel and OpenVINO to reduce memory usage and increase inference speed for small models such as SmolVLM.

Key Steps:

1. Model Conversion The model needs to be converted into OpenVINO IR format. You can either use optimum-cli or convert it on the fly at the time the model is loaded.

2. Quantization This process lowers the model's precision to reduce its size and increase speed, and there are two options.

• Weight Only Quantization (WOQ): Quantizes only the Weights. This improves memory efficiency and loading speed, but since the Activations retain their original precision, the improvement in inference speed is limited. The advantage is that accuracy loss is minimal.
• Static Quantization: Quantizes both the weights and the activations. This requires a Calibration step using a representative dataset to find the optimal parameters, and it yields a larger performance improvement.