Open LLM Leaderboard Carbon Emissions Analysis

HuggingFace analyzed carbon emissions generated during inference for over 3,000 models evaluated on the Open LLM Leaderboard. Emissions were calculated using a heuristic method that factors in evaluation time, hardware power consumption, and the carbon intensity of the power source.

The key findings of the analysis are as follows:

• Model Size and Emissions: As model size increases, CO₂ emissions increase, but a diminishing returns phenomenon was observed where performance gains are not proportional to the increase in emissions.
• MoE Model Efficiency: Mixture-of-Experts (MoE) models showed very long inference times for some models, resulting in relatively lower carbon emission efficiency relative to performance.
• High-Efficiency Models: Smaller models such as Qwen-2.5-14B and Phi-3-Medium were confirmed to have the best carbon emission efficiency relative to performance.

This analysis suggests the importance for model creators and users to consider not only performance but also environmental responsibility when selecting models.