Tests passed but the agent was broken

The failure point of LLM agents wasn't the code — it was tool routing.

• CI and unit tests all passed, but the actual agent produced silent misroutes, such as instructing the user to install a plugin that was already installed.
• The author calls this Agentic COBOL, and proposes feeding real natural-language prompts into a real runtime and grading whether the correct tool_use occurs.
• Python mocks, SDK boundary tests, and output-quality evaluations like RAGAS, DeepEval, and LangSmith don't catch this problem, because what needs to be verified is not the code path but the combination of LLM + tool catalog + description text.
• Each run is an actual LLM call, taking 10-30 seconds and incurring API costs and nondeterminism.
• The implementation consists of a corpus of 30 prompt JSONs, expected routes per runtime, and a runner with a streaming JSON parser for Claude Code. In this cycle, Claude Code was run on Linux x86 across two consult-provider modes (Together AI cloud, local GPU), three times each, checking 180 routing decisions with 0 failures.
• The v13 harness was expanded into a matrix of 30 prompts × 3 runtimes × 3 hosts × N model combos × N consult-provider modes, leading to the conclusion that systems where human language factors into correctness must be tested including that language layer.