Llama-2 Inference Characteristics

Llama-2-70B looks like an alternative to GPT-3.5, but based on cost and latency, it isn't always the better choice. The author summarizes that when serving Llama-2 on 2 80GB A100 GPUs, it was worse than GPT-3.5 in both price and speed for completion token generation, while it was actually more than 3x cheaper for prompt token processing.

The key is the difference in nature between prompt-heavy tasks and generation-heavy tasks. When processing a prompt, the model weights can be read once and multiple tokens can be batched for computation, making it efficient, but when generating tokens, the model weights and KV cache must be re-read for every single token, making memory bandwidth the bottleneck. So Llama-2 is well-suited for prompt-heavy tasks like classification, but inefficient for workloads that require continuously generating long answers.

To explain inference cost, the author analyzes a simplified version of the transformer's FLOPs and memory paths. For Llama-2, attention FLOPs are relatively small, amounting to only about 10.5 GFLOPs out of a total of 140 GFLOPs even at a maximum length of 8192 tokens. On the memory side, however, the cost grows during completion generation because the model weights and KV cache must be re-read for every token.

For prompt processing, efficiency is high because computation can be batched relatively well. Based on assumptions about A100 performance, the author calculates a compute requirement of about 140 TFLOPs/token, and from this derives a price of roughly $0.00042 / 1K tokens. This is much cheaper than GPT-3.5's $0.0015 / 1K tokens, and processing is estimated at about 170ms for 512 tokens and about 530ms for 1536 tokens.

Conversely, token generation is much more expensive due to the memory bottleneck. At batch size 1, it rises to as much as $0.066 / 1K tokens, and while increasing batch size lowers the cost, time to first token degrades linearly. For example, at batch size 64, the time to the first response for 512 tokens is nearly 3 seconds, and for 3596 tokens it's 20.1 seconds.

In conclusion, the cases where Llama-2 is meaningfully better than the OpenAI API are limited.

• Prompt-dominant tasks: cases like classification or large-scale preprocessing
• Batch processing: offline tasks where some latency can be tolerated
• Special cases with almost no prompt tokens: not intuitive, but can be advantageous due to memory/batch structure

For most other completion-heavy scenarios, GPT-3.5 is cheaper and faster. The author's point is clear: Llama-2 is not a general-purpose alternative, but a model whose cost efficiency only comes alive when the task type is right.