oMLX, an LLM Server for Mac, Released; SSD Cache Reduces Latency from 90s to 5s

oMLX, a native LLM inference server running in Mac environments, has been released. This project is developed based on Swift rather than Electron and is open-source under the Apache 2.0 license.

Key Performance Improvements and Features

The core technology of oMLX is a KV cache layered across RAM and SSD. It resolves the 90-second latency issue caused by recalculating the entire conversation in agent loops when using existing local models. Since the KV cache is stored on the SSD, context is maintained even after restarts, reducing response wait times to approximately 5 seconds.

• Supported Models: Text, Vision, OCR, Embedding, Reranker
• API Compatibility: Compatible with OpenAI and Anthropic APIs, allowing immediate application to existing code
• Data Security: All computations are performed on the local machine, eliminating the risk of external data leakage

Operational Considerations

Running long agent sessions with medium-sized models may require GB-scale SSD capacity. When loading multiple models simultaneously, managing memory and disk resources is crucial, and attention to cache integrity (model hash and context key) is necessary.