The Gap Between LLM Token Pricing and Actual Operating Costs

According to joint research from Microsoft Research, Stanford, Berkeley, and CMU, comparing 8 frontier reasoning models found that in roughly 1 out of every 5 cases, the actual cost of a task was higher even when the listed token price was cheaper. In the worst case, the actual cost was 28x higher than the listed price.

The main causes of this cost discrepancy are as follows:

• Differences in token consumption: For the same question, the amount of tokens used varies drastically between models. Some models used 900% more Thinking Tokens than others.
• The share of Thinking Tokens: Thinking Tokens generated during the reasoning process account for over 80% of total output cost, increasing cost volatility.
• Cost variability: Even with the same model and the same question, cost can fluctuate by up to 9.7x across different runs.

Therefore, when building LLM-based services, one should not simply consider the API's sticker price alone; instead, it is essential to measure each model's token consumption patterns and variability in order to calculate the actual cost of goods sold (COGS).