Ollama 0.32.15 Metadata Cache

The Ollama 0.32.15 pre-release version introduces a model metadata cache feature to reduce per-request overhead. The release notes do not include general latency benchmarks, so no universal speed improvement figures can be provided.

User-perceived latency is the sum of various stages beyond token generation speed, including queuing, request parsing, model discovery, metadata lookup, loading, warm-up, inference, and response transmission. If setup tasks are repeated for every request, the experience can feel slow even if the model is fast.

When measuring the performance of a local stack, the following segments should be timed separately:

• From request arrival to dispatch
• Metadata and model lookup
• Model loading or warm-up
• Time to First Token (TTFT)
• Generation and response completion

You should fix the model, quantization, prompt, hardware, concurrency, and client settings to compare cold/warm paths. Since the cache may only shorten certain segments, separate testing for invalidation and stale data behavior is also necessary.