Do agents.md files actually help coding agents?

The research paper "Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?" analyzed whether repository-level instruction files (AGENTS.md, CLAUDE.md, etc.) provide substantial help to coding agents in performing tasks.

The study evaluated agents under three conditions—no context file, LLM-generated context, and developer-written context—using SWE-bench Lite and AGENTBENCH.

Key findings:

• Success rate: LLM-generated context files showed only marginal improvement in success rate compared to having no context file at all. However, developer-written context files performed better than LLM-generated ones.
• Efficiency: Surprisingly, having no context file turned out to be the cheapest and most efficient option.

Cause analysis: While agents do follow the instructions in context files, this leads them to run more tests, search and read more files, and use more tools—ultimately increasing their workload. In other words, context files increased the thoroughness of the work, but this did not necessarily translate into a higher success rate.

Conclusion: Context files should be short and specific, and it is recommended that they have a hierarchical structure that directs agents to reference specific files as needed.