AI Briefingomlx: 30x Faster GLM-5.2 Prefill on M3 Ultra
jundot/omlx
About the project
Local LLM servers often require a trade-off between convenience and control. oMLX is designed to allow direct adjustment of model loading, context limits, and memory allocation from the macOS menu bar. Frequently used models are pinned to memory, while heavier models are automatically swapped out when needed.
The core feature is a two-tier KV cache structure spanning RAM and SSD. Frequently used blocks are kept in RAM, and offloaded to SSD when capacity is reached. Even after restarting the server, previous contexts remain in the cache, allowing requests to be processed without recalculation. This is advantageous for maintaining long conversation contexts when using tools like Claude Code.
It runs text LLMs, VLMs, OCR, embeddings, and rerankers simultaneously on a single server. Memory usage is finely controlled through LRU-based automatic swapping, manual pinning, and per-model TTL settings. It provides an OpenAI-compatible API, so you can connect without modifying existing client code.
It runs on Apple Silicon (M1~M4) and macOS 15.0 or higher. Performance differences are significant for specific model families like GLM-5.2 and MiniMax M3 when using native kernel builds. On an M3 Ultra, GLM-5.2 prefill speed is approximately 30x faster compared to the standard path. It is suitable for developers who want to run coding agents locally or reduce cloud dependency.
jundot/omlx
LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar
Python
This introduction was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report errors, attribution issues, or removal requests via Contact.