Is Grep All You Need? How Agent Harnesses Are Reshaping Agentic Search

As LLM agents' complex workflows have recently increased, research has been conducted on how retrieval strategy interacts with agent architecture and tool-calling methods.

The researchers compared the performance of Grep (keyword-based search) and Vector Retrieval through two experiments.

Experiment 1: Comparing Retrieval Methods and Tool-Calling Styles

• Using the LongMemEval dataset, tests were conducted on Chronos (a custom agent harness), Claude Code, Codex, and Gemini CLI.
• The results showed that even within the same data environment, the Grep method recorded overall higher accuracy than vector search.
• Additionally, performance differences varied significantly depending on how tool results were passed to the model (inline vs. file-based).

Experiment 2: Testing Resilience to Distraction

• By progressively mixing irrelevant conversation history into the search results, the researchers measured how well the models performed under conditions with many distracting elements.
• As a result, they confirmed that retrieval accuracy is strongly determined not just by the search algorithm itself, but by the harness used and the tool-calling style.