AI Briefingllama-swap adds matrix feature
Key point
llama-swap has released a matrix that controls which combinations of models can run simultaneously.
Details
Previously, a model could belong to only one group, but now you can define multiple run combinations with matrix.
- Run large models on their own
- Group STT + large models together
- Set up a separate combination for RAG use as well.
When a request comes in, the solver selects the lowest-cost combination based on the models currently running. Cost is calculated as the sum of evict_cost for running models that fall outside the candidate combination, and ties are broken by definition order.
[a, b, c] follows subset semantics, which also allows subsets. Only the requested models are started, and the rest are not preloaded. matrix and legacy groups cannot be used together, and any model not included in any combination can only run standalone.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.