Hugging Face Introduces LLM Leaderboard

When building LLM applications, not only quality but also speed (Latency/Throughput) and cost (Price) are very important decision factors. Artificial Analysis has introduced a leaderboard on Hugging Face that comprehensively evaluates quality, price, and speed across more than 100 serverless LLM API endpoints.

This leaderboard provides the following key metrics to help AI engineers choose the right model and API provider:

• Quality: A quality index combining MMLU, MT-Bench, HumanEval, and Chatbot Arena rankings
• Context Window: The maximum number of tokens a model can process at once
• Pricing: Price per input/output token and blended pricing
• Throughput: Median and percentiles (P5, P25, P75, P95) of tokens per second (TPS)
• Latency: Median and percentiles of time to first token (TTFT)

Tests are conducted across various workload environments, including prompt lengths (100, 1k, 10k tokens) and the number of parallel queries (1, 10). All API endpoints undergo 8 tests per day, and the leaderboard is updated based on the median of measurements from the past 14 days.