llama.cpp Introduces Dynamic Model Management Feature

Router mode has been introduced to the llama.cpp server, enabling multiple models to be dynamically loaded, unloaded, and switched without restarting the server. This feature uses a multi-process architecture that runs each model in a separate process, so that a crash in one model does not affect the others.

The key features are as follows:

• Auto-discovery: Automatically finds GGUF files in the llama.cpp cache or a specified directory.
• On-demand loading: Models are automatically loaded into memory when first requested.
• LRU eviction: When the configured maximum number of models (--models-max) is exceeded, the least recently used model is automatically unloaded to free up VRAM.
• Request routing: Routes requests to the appropriate model based on the model field in the API request.

Users can manually load or unload models via the HTTP API, and can also easily switch models through a dropdown menu in the built-in Web UI. In addition, the presets feature allows different settings, such as context size or temperature, to be specified per model.