AI BriefingCross-Platform MoE Dispatch Kernel Based on Triton Released
Key point
Using OpenAI Triton, a high-performance MoE inference kernel supporting both NVIDIA and AMD has been developed.
Details
A new MoE (Mixture-of-Experts) inference kernel called TritonMoE, written in OpenAI Triton, has been released. This kernel offers high portability, capable of running on both NVIDIA and AMD GPUs without depending on a specific vendor's CUDA.
As a core technology, the Fused gate+up GEMM approach has been introduced. This computes the SwiGLU projection together in shared tile loads, reducing global memory traffic by 35%.
In terms of performance, it showed 89-131% throughput compared to Megablocks on A100, and the same kernel can also run on AMD MI300X without modification. However, it has a characteristic of degraded performance when the token count exceeds 2048 or when 64 or more experts show extreme routing imbalance.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Apple Silicon-only inference engine 'Lily' improves decoding speed by 35% over MLX-LM
Step 5 Preview Released: 600B Parameter MoE Model Highlights Agentic Tasks and Financial Analysis Performance