Mixture-of-Kittens: Open-Source MoE Megakernel for NVL72

Cursor has open-sourced Mixture-of-Kittens (MoK), which it has been using to train its agentic coding model Composer. MoK is an MoE training megakernel designed for NVL72 environments, integrating MoE communication and computation into a single deterministic kernel.

MoE layers were a major bottleneck that could account for more than half of Composer's training time. Existing implementations handled inter-GPU token communication and expert FFN operations separately, but in production environments, communication costs grew as large as computation costs, limiting overall performance.

GB300 NVL72 is a multi-node rack where 72 GPUs are connected within a single NVLink domain. MoK leverages this structure to finely overlap communication and computation, and also reduces CPU work and CPU-GPU synchronization costs caused by the integrated Grace CPU being slower than the GPUs.

The DSV3-style MoE layer addressed by MoK consists of one shared expert and multiple routed experts. When the router assigns each token to the top k experts, the token moves to the GPU where the expert is placed, undergoes FFN computation, and the result returns to the original GPU.

Key design directions are as follows:

• Integrating all MoE communication and computation into a single megakernel
• Improving GPU utilization through overlapping communication and FFN computation
• Supporting MXFP8 training
• Ensuring deterministic execution
• Minimizing CPU-GPU synchronization using ring token buffers

In comparisons based on the public implementation, MoK increased MXFP8 forward throughput on GB300 NVL72 by up to 2.37x. In actual training stacks using multiple NVL72 racks, the total number of tokens processed per second increased by 1.41x.