AI BriefingvLLM offers PyTorch-native TPU backend for Google Cloud
·2026.10.20 07:34
Key point
The new vllm-torchtpu plugin allows standard PyTorch models to run on TPU v7x with no code changes, using custom JAX/Pallas kernels for attention and quantization.
Details
- vLLM TorchTPU is an out-of-tree platform plugin that enables vLLM to serve PyTorch models on Google Cloud TPUs without re-implementing model code in JAX.
- The backend utilizes TorchTPU (torch.compile(backend="tpu")) and custom JAX/Pallas kernels for Ragged Paged Attention, Mixture-of-Experts, and FP8/INT8/MXFP4 quantized matmuls.
- TPU v7x (Ironwood) is the recommended hardware, validated in CI on every pull request and nightly, while TPU v6e, v5e, and v5p are currently in experimental status.
- The plugin supports standard vLLM features including speculative decoding, prefix caching, multi-LoRA, and disaggregated prefill/decode serving across single and multi-host TPU slices.
- Installation requires Python 3.12 and is available via PyPI or pre-built Docker images, with first-class support for models like Qwen3, DeepSeek-V3, Kimi-K2.5, and Llama 3/4.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.