John Hennessy et al. Publish Paper on Intelligence per Watt (IPW) Efficiency of Local LLMs

Researchers including John Hennessy and Christopher Ré proposed Intelligence per Watt (IPW), a metric that holistically measures the energy efficiency and performance of local LLMs, and released a large-scale empirical study based on it.

Research Background and Metric Definition

To alleviate the burden caused by increasing demand for centralized cloud LLM infrastructure, the study explored the viability of utilizing small local LLMs (with active parameters of 20B or less) and local accelerators (e.g., Apple M4 Max). IPW is defined as task accuracy per unit of power, serving as a key metric to simultaneously evaluate the capability and efficiency of local inference.

Key Experimental Results

The analysis of 1 million real-world single-turn chat and reasoning queries across more than 20 state-of-the-art local LLMs and 8 types of hardware accelerators yielded the following results:

• Performance Coverage: Local LLMs successfully answered 88.7% of all queries, with accuracy varying by domain.
• Efficiency Improvement Trend: A longitudinal analysis from 2023 to 2025 showed a 5.3x improvement in IPW. Driven by advancements in algorithms and accelerator technology, the coverage of locally processable queries surged from 23.2% to 71.3%.
• Hardware Comparison: When running the same model, local accelerators measured at least 1.4x lower IPW compared to cloud accelerators, suggesting that there is still room for optimization in local hardware.

Implications

Local inference can meaningfully offload central infrastructure demand for a significant proportion of queries, and IPW can serve as a crucial metric for tracking this transition.