How LLM Inference Works

Text is first tokenized, and most LLMs use Byte Pair Encoding (BPE) to split sentences into roughly 50,000 token pieces. Each token is then turned into a 4,096-dimensional vector via an embedding table, and positional information such as RoPE is added to reflect order.

Next, a chain of transformer layers operates in sequence. Each layer mixes information between tokens via self-attention, processes the representation through a feed-forward network, and then projects the vector at the last position onto the vocabulary size to produce the next-token probabilities.

Inference actually splits into two stages: prefill and decode. Prefill, which processes the entire prompt at once, is a compute-bound phase that computes Q/K/V for all tokens in parallel, while decode, which happens after the first token is produced, is a memory-bound phase that computes only one new token at a time. The former governs TTFT (Time to First Token), and the latter governs ITL (Inter-Token Latency).

The key optimization is the KV cache. Storing the K/V of previous tokens to avoid recomputation can boost speed by more than 5x on long responses, but the cache consumes GPU memory rapidly. Per the article, a 13B model uses roughly 1MB per token, and a 4K token context uses about 4GB of VRAM just for the cache.

There are also various mitigations.

• INT8/INT4 quantization reduces weight memory and boosts throughput compared to FP16/BF16.
• GPTQ and AWQ reduce quality loss through per-channel scaling.
• Grouped-query attention and PagedAttention reduce the K/V memory burden or manage it in a page-like manner.

The difference is clear just by looking at weight sizes. A 7B parameter model drops to roughly 28GB in FP32, 14GB in FP16, 7GB in INT8, and 3.5GB in INT4, and unlike training, inference is far more tolerant of such lower precision. This makes it possible to run on smaller GPUs or laptop-class environments.

The DeepSeek V4 family also demonstrates an approach of shrinking the cache itself. It's presented as reducing cache size to about 10% and per-token computation to about 27% at a 1 million token context, and vLLM, TensorRT-LLM, and Text Generation Inference use continuous batching and speculative decoding to batch multiple users together on the same GPU. Generated tokens are converted back into characters and streamed.

The practical implications are clear.

• Long prompts worsen TTFT, and long outputs worsen ITL.
• Extending context grows the cache and reduces batching efficiency.
• Quantization is the biggest lever, and speed shouldn't be judged by GPU utilization alone.

Ultimately, perceived speed depends more on memory bandwidth and cache design than on the amount of computation. When a model is slow, the first question to ask isn't whether you need a stronger GPU, but whether the start is slow or the streaming is slow.