Gap Confirmed Between RTK's Claimed Token Savings and Actual Cost Benchmarks

An analysis has revealed that the token savings effects promoted by RTK (Rust Token Killer), a terminal output compression tool for AI agents, do not align with actual cost benchmark results. Experiments using Terminal-Bench 2.1 showed that cost savings from applying RTK were either not statistically significant or resulted in increased costs.

Benchmark Results: Lack of Cost Savings

Comparing costs and performance with and without RTK in Claude Code (Fable 5.0) and OpenCode (DeepSeek V4 Pro) environments revealed the following patterns:

• Claude Code + Fable 5.0: Applying RTK reduced total costs by 5% ($596→$546), but the average cost change per task was +1%, which was not statistically different from zero. The Pass rate dropped by 1%, from 84% to 83%.
• OpenCode + DeepSeek V4 Pro: Applying RTK increased total costs by 5% ($26→$31), and the average cost per task increased by 17%. The Pass rate dropped by 2%, from 71% to 69%.
• Lack of Consistency: For Fable, savings were concentrated in specific tasks (winning-avg-corewars), with savings rates below 1% for the remaining tasks. For DeepSeek, the average cost increased by 18% even in tasks where all attempts passed.

Metric Limitations and the 'Tokenflation' Issue

The rtk gain metric reported by RTK diverges significantly from actual billed costs. This metric is calculated based on the byte difference between original and filtered outputs but fails to reflect additional costs caused by an increase in the number of agent turns.

• Metric Distortion: In the DeepSeek experiment, rtk gain reported an 89% token reduction, but actual task costs did not decrease. This is due to a calculation method unrelated to the actual cost structure, where savings from simple command calls like head -1 accounted for 69% of the total metric.
• Turn Count Increase: Applying RTK leads to an increase in the number of agent turns, causing a 'Tokenflation' issue where costs rise. In the DeepSeek experiment, the average input tokens per turn decreased by 7%, but the total number of turns increased by 18%, resulting in no reduction in total input volume.
• Ignoring Cache Effects: In modern agent coding environments, context is cached, making re-read costs low. Since Fable's cache read cost is 1/10 of standard input and DeepSeek's is 1/30, the economic benefit of terminal output compression is limited.

Conclusion and Implications

Latest frontier models use terminals efficiently (e.g., only about 7% of Fable's context is terminal output) and utilize their own output limiting features like head/tail. RTK was likely more effective with older models but is not recommended as a general-purpose cost-saving tool in the current environment. Additionally, RTK only rewrites the Bash tool and bypasses Read, Grep, etc., limiting its scope of application.