Apple Silicon-only inference engine 'Lily' improves decoding speed by 35% over MLX-LM

The lightweight inference engine Lily, optimized for the Apple Silicon architecture, significantly improved performance over MLX-LM. Running the Qwen3.6-35B-A3B model in an M5 Max (40-core GPU, 128GB RAM) environment, Prefill speed improved by an average of 1.23x, and Decode speed improved by an average of 1.35x (35%).

Lily applied the following optimizations tailored to the Qwen3.6-35B-A3B model, which features MoE routing and a Gated DeltaNet structure.

1. Prefill Optimization: Dequantization of 4-bit weights was performed inside Grouped GEMM, eliminating memory writes and reads of intermediate arrays. In experiments comparing this configuration with one using separate dequantization operations, end-to-end prefill throughput for a 512-token prompt improved by 77.4%. An additional optimization was applied to keep expert routing on the GPU, eliminating CPU synchronization.
2. Decode Optimization: The 'GPU Token Handoff' technique minimized data round-trips between CPU and GPU in a batch size of 1. Additionally, for long contexts (32K and above), switching to a fixed block layout improved end-to-end decoding speed by 40.2% at a 128K context.

Performance analysis showed that MoE GEMM reached 97.9% of the maximum sustained weight read speed measured for this access pattern. Since weight reading remained the bottleneck, there was little room for further optimization. However, Speculative Decoding was not applied as it actually reduced speed by 18% due to inefficient operation patterns and increased expert weight reads. Lily recorded superior throughput compared to MLX-LM across all measured lengths, while maintaining numerical consistency (Perplexity difference of 0.04%).