Running a 35B model on a Mac in real use

For local Mac use, SwiftLM published benchmarks running Qwen 3.6 35B with up to 100K context.

On a 24GB Mac, by streaming layer weights from NVMe SSD and applying TurboQuant KV cache compression, they fit the 35B dense model into memory at a 23.2GB footprint at 40K context.

However, speed dropped at long context, falling to around 2.5 tok/s, and by attaching a 0.8B speculative draft decoder, they reported recovering to 7.4 tok/s, about a 3x improvement.

They also explained that while natively mapping 100K context on M5 Pro requires 63.9GB, the combination of SSD streaming and TurboQuant reduced the memory requirement by about 60%, achieving a 28.3GB load.