#prefix-caching

The latest AI and developer news about #prefix-caching, with the original source and a short summary.

Qwen3.8 27B | 1 x R9700: 262K context, half a million tokens of reusable cache, ~180 tok/s. And yes, let's talk about the "3-bit" :)

New mixed-precision 3-bit weights for Qwen3.8 27B enable a 569,878-token reusable cache on a single AMD R9700, accelerating agent workflows by up to 12x for first-token latency with a 2.9-point MMLU-Pro drop vs MXFP4.

·2·2026.10.04 22:00

Optimizing Gemma 4 Serving with vLLM on Amazon EKS [Part 2: Throughput and Latency SLOs on a Single GPU]

This article introduces vLLM optimization strategies to maximize Gemma 4's SLO-compliant throughput through NVFP4 precision and the separation of interactive and batch pools.

·2026.09.10 09:00

Predictive Speculative KV Replication for Bursty LLM Inference

BTB predicts repeated-prefix bursts and proactively replicates KV caches into GPU HBM.

·2026.08.01 04:00

Report: GKE Inference Gateway Boosts AI Response Speed by Up to 92%

GKE Inference Gateway has dramatically improved LLM inference performance through Prefix Caching and model-aware routing.

·2026.06.10 01:00

Powering Agents: Workers AI Starts Supporting Large Models Beginning with Kimi K2.5

Cloudflare Workers AI has added Kimi K2.5, supporting a 256k context window and tool calling.

·1·2026.03.20 04:00

SK Planet Shares Development Story of B2B Video Generation Service 'AI Moment' Combining GPT-4o and TTS

SK Planet shared the development experience and optimization strategies behind 'AI Moment,' a B2B service that automatically generates corporate advertising videos using GPT-4o and Google TTS.

·1·2025.05.29 09:00