LLM Output Compression Cuts Costs by 3x

A study comparing the cost and accuracy impacts of input prompt compression and output instruction compression across 9 LLM models has been published.

Effects of Output Compression

• Cost Reduction: Reducing output tokens lowered API costs by an average of 1.5x, and up to 3x.
• Accuracy Maintenance: The accuracy of shortened outputs remained nearly identical to the unconstrained baseline.
• Multilingual Application: Similar cost reduction effects were confirmed across 11 languages, including English, German, and Japanese.

Counterproductive Effects of Input Compression

• Cost Increase: Shortening input prompts caused models to generate longer answers to fill in missing information, resulting in cost increases of up to 96% in the worst-case scenarios.
• Accuracy Degradation: A decline in answer accuracy was observed when inputs were shortened.

Key Takeaways

• Since output tokens are more expensive than input tokens, prompting that reduces the number of output tokens is effective for cost savings in short, single-turn tasks.
• However, in about 50% of cases, the model's original reasoning process and text may not match when output is shortened, but this is not a major issue if only the final answer matters.