The Pitfalls of Ollama KV Cache Configuration

The OLLAMA_KV_CACHE_TYPE setting in Ollama is not applied individually per model but is applied globally to all models loaded on the server. While this setting is used to improve memory efficiency, the documentation states that there is no per-model override capability.

Performance and Memory Impact by Quantization Method

• f16: The default value, offering high precision and high memory usage.
• q8_0: Uses approximately half the memory of f16, with generally negligible quality loss.
• q4_0: Uses approximately one-quarter of the memory of f16, but the quality loss is not insignificant. Quality degradation becomes more pronounced as context length increases.

Imbalance of Impact Across Models

The impact of cache quantization varies depending on the model architecture. According to the official Ollama documentation, models with a high GQA (Grouped Query Attention) count may experience greater precision loss than those with lower counts. The documentation explicitly cites the Qwen2 series as an example of models with high GQA counts. Therefore, the phenomenon where only some models suffer performance degradation under the same global setting is not a random error but stems from structural characteristics.