AI BriefingLLM Performance Optimization: Understanding Prefill/Decode
Key point
This explains the mechanisms and key performance metrics of Prefill and Decode, the core stages of LLM inference.
Details
Token generation in LLMs has an auto-regressive characteristic that depends on previous tokens. This process is broadly divided into two stages.
The Prefill stage is the process of processing all tokens of the input prompt in parallel to generate the first output token. During this stage, the KV Cache is created, and since all input tokens can be computed at once, parallelization is possible.
The Decode stage is the process of generating tokens after the first token. Since each token depends on the results of previous tokens, parallel processing at the individual request level is not possible, and it proceeds sequentially.
The key metrics for measuring LLM performance are as follows:
- Time to First Token (TTFT): The latency of the Prefill stage, meaning the waiting time until the user sees the first response.
- Time per Output Token (TPOT): The latency of the Decode stage, meaning the speed at which a single token is generated.
For conversational services, to ensure a smooth user experience, it is recommended to maintain TTFT within 3 seconds and TPOT at 100~300ms (3~10 tokens per second).
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
KV Cache Research: Limitations of LRU Alternatives and Failure in Capacity-bound Environments
LLMTraceFX Introduces Independent Auditor to Verify That Cache Hits Do Not Prove Computation OmissionSalesforce Releases 'Random Attention' KV Cache Eviction Policy for Reasoning Models