Luce Spark Released, Running a 35B MoE on a 16GB GPU

Luce Spark is an open-source tool that enables efficient execution of MoE (Mixture-of-Experts) models in the 33B–35B range on a 16GB GPU environment.

Unlike conventional simple offloading methods, it learns the routing patterns that occur during actual inference, keeping frequently used 'Hot' experts on the GPU while leaving the rest in system RAM, swapping them in only when needed. It optimizes in real time without any separate pre-calibration, and the learned profile is updated upon model restart.

Key performance and features:

• High efficiency: On an RTX 3090, with only 60% of experts resident on the GPU, it achieves about 100 tok/s, which is 1.5x faster than the conventional simple offloading approach (66 tok/s).
• Real-time optimization: Learns expert placement based on real-time routing data without any separate corpus.
• Open source: Released on GitHub under the Apache 2.0 license.