Ollama VRAM Shortage Caused by Default Settings

In Ollama, the phenomenon where models are not fully loaded onto the GPU and are offloaded to the CPU is often due to Default Settings rather than issues with the model itself. If the PROCESSOR status shown by the ollama ps command is not 100% GPU, you should check the following three default settings.

1. Context Window Smaller Than Expected

Ollama's default context length is 4096 tokens. Even if a user specifies a 128k model, only 4096 tokens are used unless num_ctx is explicitly set. If the prompt exceeds this range, the beginning is truncated, which is not a degradation in model performance but a result of the input not fitting within the window.

2. Memory Multiplication Due to Parallel Requests

Memory requirements are calculated as OLLAMA_CONTEXT_LENGTH × OLLAMA_MAX_PARALLEL_REQUESTS. When the default number of parallel requests is 1 or more, increasing the context length causes memory usage to increase multiplicatively. For example, setting 4 parallel requests for a 2K context allocates memory equivalent to an 8K context, which can lead to VRAM shortages.

3. VRAM Retention of Unused Models

The default value for OLLAMA_MAX_LOADED_MODELS is 3 times the number of GPUs (or 3 for CPU inference), and models remain in VRAM for 5 minutes after their last use. As a result, an environment that worked normally in the morning may experience spilling in the afternoon due to residual memory from other models. Memory can be immediately released using ollama stop.