RunAnywhere Open-Sources NKI Megakernel Inference Engine for GLM-5.3-Flash on AWS Trainium2

Details

RunAnywhere open-sourced an inference engine for AWS Trainium2 that serves GLM-5.3-Flash (320B MoE, 1M-token context) on a single trn2.48xlarge instance.
Batch-8 decode latency improved from 32.2 ms to 24.2 ms between October 17 and November 1, driven by fused sparse-attention layers and KDA/MoE kernels that each saved 3.2 ms per step.
Profiling revealed that in early builds, only 2% of a decode step was actual model math, with 36% spent on mHC pre-projection and 33% on XLA glue, prompting a rewrite to keep working sets on-chip and overlap collectives.
End-to-end performance on the AgentX benchmark reached up to 28.9 agent requests per minute, with a median decode speed of 35.9 tokens/s at 16 agents.
Accuracy is maintained via byte-for-byte checks and high-precision references, with a remaining approximation in sparse-attention prefill affecting 1.2% of selections and shifting log-probabilities by 0.012 to 0.021 nats on average.

Key point