AI BriefingMiniMax-M2.7, 5090 Mixed Configuration Experiment
Key point
In an RTX 5090 Llama.cpp experiment, MiniMax-M2.7 processed at 22 tps even at 100k context but only generated at 3~4 tps.
Details
To see whether slow generation and compaction delays could be offset by running things overnight, the MiniMax-M2.7 229B quantized model was tested with Llama.cpp on an RTX 5090 32GB VRAM + 64GB RAM setup.
For a search-based research workflow, 60k context was judged necessary, and under 40k was deemed risky. A harness like Hermes was estimated to use 10k context just to start, plus 10k context needed per search result.
The initial configuration was --fit-ctx 40000, --no-mmap, --parallel 1, and the first run used 18 GPU layers / 45 CPU layers, 100k context, along with SSD mmap-based progressive weights loading.
The results were as follows.
- On a BF6 release-date query with a search tool attached, results were truncated, leading to an incorrect answer.
- A Hermes-based collection of 24-hour trending topics from the localLLaMA subreddit timed out after over 30 minutes.
- Processing speed was 22 tps, and generation speed was 3~4 tps.
The conclusion reached was that using SSD as extended memory is not practical.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Underdog Releases 'Husky', a Model-Specific Inference Engine Up to 4.5x Faster than MLXNVIDIA Open-Sources Local AI Inference Router 'PAIR'