SageMaker Llama 2 Deployment Benchmark

Hugging Face announced comprehensive benchmark results for efficiently deploying the Llama 2 (7B, 13B, 70B) models in the Amazon SageMaker environment.

Using the Hugging Face LLM Inference Container (based on TGI), they analyzed over 60 deployment configurations. Various EC2 instance types and load levels were tested to measure Latency and Throughput.

Optimization strategies by use case are as follows:

• Best cost efficiency: Achieving adequate performance at low cost
• Lowest latency: For services where real-time response is critical
• Best throughput: Maximizing tokens processed per second

It also covers optimization methods using GPTQ quantization to reduce model size so it can run even on a single GPU. All experiment code and data are publicly available on GitHub.