Core Technologies for LLM Inference Optimization: Quantization, KV Cache, and Inference Chips

With the proliferation of agentic AI, the AI industry's focus is shifting from training to inference optimization. Deloitte forecasts that the share of inference in total AI computing will grow from about one-third in 2023 to about two-thirds by 2026, and predicts that inference will account for approximately $20.6 billion of AI cloud infrastructure spending in 2026, surpassing training spending for the first time.

Quantization is a technique that reduces model size by converting FP16 weights to lower-precision data types like INT4. Switching from FP16 to INT4 can reduce model size by approximately 75%, allowing larger models to run even with limited VRAM, while also reducing memory access and computational complexity to lower response latency.

However, as context length and concurrent users increase, the KV cache (Key-Value Cache) emerges as a new bottleneck. The KV cache stores the state of already processed tokens to prevent redundant computation, but memory usage increases linearly with context length. Even if model weights are reduced, the KV cache can fill up GPU VRAM, and existing quantization methods incur a 1-2 bit overhead for storing per-block quantization constants.

TurboQuant from Google Research compresses the dynamically growing KV cache itself, rather than model weights. It uses PolarQuant to transform data vectors, eliminating the need for normalization and quantization constant storage, and uses QJL (Quantized Johnson-Lindenstrauss) to reduce high-dimensional data to 1-bit signs, minimizing additional memory overhead.

In Google Research benchmarks, TurboQuant demonstrated the following performance:

• Reduced KV cache memory by at least 6x in long-context tasks
• Accelerated attention logit computation by up to 8x on H100 GPUs with 4-bit settings
• Quantized the KV cache down to 3 bits without additional training or fine-tuning
• Achieved higher 1@k recall in GloVe vector search compared to existing quantization techniques

Hardware is also being restructured around inference. Inference-specialized processors like the Groq LPU (Language Processing Unit) and Google TPU are gaining attention as alternatives to GPUs optimized for large-scale training, targeting real-time generation and cost efficiency. Ultimately, the competitiveness of agentic AI infrastructure depends not just on model size, but on jointly optimizing weights, KV cache, and inference hardware.