Recipe Released for Serving 320B GLM-5.3-Flash on Free Kaggle TPUs

Developer Abdur Rahim released the open-source repository kaggle-tpu-lab, which serves the GLM-5.3-Flash (320B MoE) and Qwen3.8-27B models on Kaggle's free TPU v5e-8 environment and generates OpenAI/Anthropic-compatible APIs.

GLM-5.3-Flash Serving Technology

Since the existing vLLM TPU backend could not run GLM-5.3-Flash due to the lack of Linear Attention kernels, the author developed a custom inference engine based on JAX/Pallas with approximately 5,400 lines of code.

• Quantization Strategy: Expert weights are quantized to 3-bit and the rest to int8, optimized to run on eight 16GB-class TPU chips.
• Performance: Achieved approximately 64 tok/s for a single stream and approximately 90 tok/s for multiple (3) streams, supporting a context length of 262,144 tokens.
• Limitations: There is a difference in output quality compared to the original bf16 model due to 3-bit quantization, and no formal benchmark comparison was conducted.

Qwen3.8-27B and vLLM Patch

Qwen3.8-27B runs with bf16 weights as-is by applying a patch to vllm-tpu.

• MTP Bug Fix: Applied a patch (porting PR #3178) to resolve the issue where vllm-tpu v0.28.0 failed to roll back the recurrent state when rejecting MTP draft tokens.
• Performance Improvement: Achieved 100% greedy decoding match and a 34% improvement in decoding speed (increased from 78 tok/s to 104 tok/s based on A/B testing, approximately 130 tok/s in deployment settings).

Usage Conditions and Precautions

• Quota: Uses approximately 20 hours of free TPU quota per week with a phone-verified Kaggle account.
• Session Limit: Kaggle sessions are limited to 9 hours, and the endpoint address changes upon restart.
• Network: Uses a cloudflared tunnel; keep-alive at 15-second intervals and streaming responses are essential to bypass Cloudflare's 100-second timeout.
• License: The repository is under the MIT license, and model weight licenses follow Apache-2.0 (Qwen) and MIT (GLM) respectively.