Coding Agent Research: Preference for grep over LSP and the Importance of Tool Interface Design

Research results analyzing why coding agents prefer lexical search (grep) over semantic navigation (LSP) and how tool interface design affects agent performance have been released. According to the study, although LSP can improve precision, agents more frequently choose the familiar grep path, and success rates actually decreased when its use was enforced.

grep Preference and LSP Limitations

• Tool Selection Patterns: In simple code location tasks, the selection rate for semantic tools was only 0–6%, and success rates dropped from 100% to 89% when semantic tools were enforced. In contrast, for reference completeness tasks (finding all callers), the selection rate for semantic tools rose to 45–57%.
• Precision and Recall: In reference completeness tasks, the precision of LSP-based paths was 1.00, higher than grep (0.76), but recall was similar for both paths at approximately 0.66. This indicates that semantic navigation does not find more actual calls.
• Impact of Codebase Noise: The advantages of LSP become clear in codebases where grep precision is low (high noise). For example, in a noisy TypeScript repository (hono), using LSP improved the F1 score by +0.246 and reduced token costs by 12%. Conversely, in a clean repository (remeda), LSP acted as pure overhead, increasing token costs by 16%.

Importance of Tool Interface Design

• Impact of Output Format: When LSP tools were changed to return source text context along with location information, rather than just location information, Pass@1 for Multi-file rename tasks increased from 0.67 to 0.83. Additionally, the number of subsequent file reads dropped significantly from 15.2 to 3.2 per episode.
• Role of the Harness: Model capability is defined as the product of the model itself and the runtime surrounding it (the Harness). Since tool input schemas, result formats, and error handling methods determine the agent's behavioral policy, new search interfaces must be evaluated as part of the entire agent loop.

Implications

When building agent platforms, models and Harnesses must be evaluated together, as more sophisticated abstraction does not always guarantee better performance. It is important to verify success rates and token efficiency in real tasks and measure whether agents actually invoke tools (adoption rate). Since semantic search and lexical search solve different problems, a routing strategy based on codebase characteristics and task types is necessary.