MLX Enables Frontier LLM Inference on Mac Studio Clusters via RDMA

Apple's MLX framework now supports distributed large language model inference on Mac Studio clusters, enabling the execution of frontier-scale models locally. By leveraging macOS 26.2's new Thunderbolt 5 RDMA drivers and MLX v0.30.1+, users can cluster multiple Mac Studios to run models that exceed the memory capacity of a single machine.

Hardware and Software Stack

The performance gains rely on specific hardware and software updates:

• Hardware: Mac Studio with M5 Ultra chips (up to 512GB Unified Memory) and previous generation M3 Ultra.
• Software: MLX v0.30.1+ introduces the jaccl communication collective, which utilizes RDMA for low-latency multi-node operations via mlx.launch.
• Optimization: The MLX_METAL_FAST_SYNCH=1 environment variable reduces CPU-GPU synchronization overhead.

Performance Benchmarks

Benchmarks were conducted using mlx_lm.benchmark on three models: Muse Spark Open (816B total/42B active parameters, MoE), Mistral Medium 3.5 (128B, Dense), and Qwen 3.8 (27B, Dense). The M5 Ultra demonstrates significant improvements over the M3 Ultra:

• Time to First Token (TTFT): Up to 4.6x faster due to the GPU's Neural Accelerators.
• Decode Speed: 1.6x faster thanks to increased Unified Memory bandwidth (1.2 TB/s vs 819 GB/s).
• Power Consumption: Approximately 300W for M5 Ultra vs 200W for M3 Ultra at 65,536 context length.

Distributed Inference Strategies

Two parallelization strategies were evaluated for multi-Mac setups:

1. Pipeline Parallelization: Splits the model by depth. It is simple to implement and has low communication overhead. For Muse Spark Open, a 2-node pipeline setup achieved a TTFT of 8.4s at 8,192 context.
2. Tensor Parallelization: Splits the model by width. It requires low-latency communication (jaccl) but offers better scaling for Dense models. A 4-node tensor setup for Muse Spark Open achieved a TTFT of 4.8s and a decode speed of 36.73 tokens/s.

Key Results (8,192 Context)

• Muse Spark Open:
  • 4-node Tensor: 4.8s TTFT, 36.73 tokens/s.
  • 2-node Pipeline: 8.4s TTFT, 15.74 tokens/s.
• Mistral Medium 3.5:
  • 4-node Tensor: 7.5s TTFT, 23.37 tokens/s.
  • Single M5 Ultra: 24.6s TTFT, 7.70 tokens/s.
• Qwen 3.8:
  • 4-node Tensor: 1.7s TTFT, 50.89 tokens/s.
  • Single M5 Ultra: 4.1s TTFT, 19.69 tokens/s.

Local Agentic AI

The update supports local agentic workflows via mlx_lm.server. Users can launch distributed servers using commands like mlx.launch --hostfile hostfile.json --backend jaccl to serve models like Muse Spark Open on port 8080, compatible with tools such as OpenCode.