Coding Agent Tool Selection Analysis: Claude Code Relies on Prior Knowledge, Codex Leverages Web Search

Results from an analysis of tool selection patterns in major coding agents such as Claude Code, Codex, and Cursor, based on 75 repositories and 1,163 variant tasks, have been released. Out of 16,893 total runs, 5,292 valid sessions were analyzed, revealing distinct differences in information sources and decision-making methods among the agents.

Information Sources and Selection Tendencies by Agent

• Claude Code: Primarily relies on the model's prior knowledge, performing web searches in only about 30% of cases. However, in cutting-edge areas such as sandboxes, the web search rate rises to 80%. Its in-house construction rate is 19%, approximately twice as high as other agents.
• Codex: Uses web search in 94% of sessions, with 90% of those focusing on trusted domains or specific solutions via the site: operator.
• Cursor: Makes web-based decisions in 2/3 of sessions.

Repository Context and Vendor Influence

Even for identical requests, the selected tools vary depending on the repository's language and context. For example, when selecting an email sending service, Resend was primarily chosen in TypeScript repositories, Sendgrid in Python, and Postmark in Go. Vercel showed a 100% win rate in TypeScript (NextJS) environments but was not recommended in Python environments.

Gap Between Mention Volume and Actual Selection

Many vendors are frequently mentioned in conversations but do not lead to actual selection. PayPal was mentioned 139 times but was never selected, and LangChain was cited 194 times but selected only 4 times. In contrast, Stripe was selected in 9 out of 10 cases, and Neon achieved a 66% win rate, securing an overwhelming advantage in their respective categories. This suggests that feature listings on vendor pages and bundle pricing policies significantly influence agent selection.