TokenSpeed: An Ultra-Fast LLM Inference Engine for Agentic Workloads

TokenSpeed is an LLM inference engine designed from the ground up for agentic coding workloads. Premised on long contexts exceeding 50K tokens and many turns, it combines compiler-based parallelization modeling, safe KV cache reuse constraints, a layered kernel system, and SMG integration to reduce even CPU-side request entry costs.

The modeling layer adopts a local SPMD structure. By placing I/O placement annotations at module boundaries, a lightweight static compiler automatically generates the necessary collective operations, so developers don't need to implement communication logic themselves.

The scheduler separates the control plane from the execution plane. The control plane uses C++'s FSM and type system to verify KV cache state transitions, resource ownership, and operation overlap timing at compile time, while the execution plane remains in Python to boost research and development speed.

The kernel layer is a pluggable subsystem separated from the core engine.

• It provides a plugin architecture, a common API, and a central registry/selection model for NVIDIA Blackwell optimization and heterogeneous accelerator support.
• The decode kernel batches q_seqlen and num_heads together to increase Tensor Core utilization.
• The binary prefill kernel fine-tunes softmax using NVIDIA internal knobs, and TokenSpeed MLA has been adopted by vLLM.

The performance evaluation was conducted based on an implementation that began in mid-March 2026. Coding agent traffic was reproduced using EvalScope and SWE-smith traces, with the goal of maximizing TPM per GPU while keeping per-user TPS typically at 70, and in some cases above 200. This release is limited to single deployments without PD disaggregation, with related support previewed for a follow-up post.

In Kimi K2.5 benchmarks, the Attention TP4 + MoE TP4 configuration proved to be the most suitable combination for coding agents. In this setup, TokenSpeed outperformed TensorRT-LLM's entire Pareto frontier, achieving roughly 9% lower latency at batch size 1 and roughly 11% higher throughput around TPS/User 100.

In another key comparison involving MLA, TokenSpeed's binary prefill kernel exceeded TensorRT-LLM's MLA across all five representative prefill workloads. In decode phases with speculative decoding, it cut latency by nearly half under conditions of batch sizes 4/8/16 and long prefix KV cache.

Development was carried out in collaboration with NVIDIA DevTech, AMD Triton, Qwen Inference, Together AI, and others, with compute support also provided by OpenAI, NVIDIA, AMD, Verda, and Nebius.