AI Briefing
KOSign in

FlashStates Improves GLM-5.3-Flash Prefill Throughput by 3.14x

·2026.11.14 18:30

Key point

In hybrid model serving, it increased prefill TPM by up to 3.14x and reduced p90 TTFT by up to 88%.

1 / 7

Details

  • Researchers from Together AI, UIUC, and UT Austin released FlashStates, a hybrid LLM serving optimization tool, as a vLLM extension
  • It stores recurrent state checkpoints at token positions rather than KV block boundaries and evaluates state and KV as a single unit to minimize cache miss recomputation
  • In a dual B200 GPU environment with 48 concurrent sessions, it achieved a 3.14x increase in GLM-5.3-Flash prefill TPM and an 88.4% reduction in p90 TTFT
  • The benefits are significant in high-concurrency intervals where the working set exceeds cache capacity, while under low load, throughput differences may be negligible or TTFT may increase
  • The code is released under the Apache-2.0 license, based on a fork of vLLM v0.33.0 and Mooncake 0.3.13.post1, with container images provided for B200/GB300
vllmllm-servinginference-optimizationhybrid-modeltogether-aiprefix-cache
Read originalShare on Threads

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.

Related

RedditPrefixwise: Open-source tool diagnoses vLLM and SGLang prefix cache misses with 19 specific causesRedditvLLM v0.33.0 Scheduler Fix Boosts DeepSeek DSpark Throughput by 2.47xLlamaIndex BlogLlamaIndex Achieves 10x Inference Speedup for OCRia-1 with Custom Optimization StackPyTorchKRHelion Linear Backend for vLLM Delivers >10% Throughput Gains on Some Hopper WorkloadsPyTorchKRPie Released: LLM Serving System That Lets User Code Control KV Cache and Generation LoopsHacker NewsNVIDIA Tutorial: Serving GLM-5.3 NVFP4 on Dual DGX Stations with vLLM