Verifying the "bash is all you need" hypothesis

In the AI community, a belief is spreading that filesystems and bash are the optimal abstraction layer for AI agents, based on the fact that LLMs were trained on massive amounts of code and terminal data.

However, Braintrust built an eval harness to check whether this hypothesis holds true even when querying structured data, and ran an experiment. The experiment compared three approaches—SQL agent, Bash agent, and Filesystem agent—on GitHub issue and PR data.

The results showed the SQL agent delivering overwhelming performance with 100% accuracy. In contrast, the Bash agent achieved only 52.7% accuracy, while consuming 7x more tokens, 6.5x more cost, and 9x more execution time. Even the basic Filesystem agent recorded 63% accuracy, outperforming bash.

What's interesting is that even though the bash agent was capable of generating highly sophisticated shell commands combining find, grep, jq, awk, and more, this did not translate into actual task performance.

The main causes of failure identified were as follows:

• Performance bottlenecks: Command execution was delayed due to things like stat() calls on 68,000 files.
• Lack of schema context: Queries failed because the structure of JSON files could not be grasped.
• Eval scoring issues: Incorrect answers resulted from dataset mismatches or ambiguous questions.