Turning the Web into a File System

AI agents' code hallucinations stem more from data problems than from problems with the model itself. Because training data can't keep up with frequent API changes and endpoint deprecations, up-to-date information is often missing.

The existing RAG approach splits documents into chunks for retrieval, so when information spans multiple pages or an exact function signature is needed, it has limits in grasping the full context.

To solve this, a method is proposed that treats web documentation like a Unix file system. It lets agents browse documentation directly using commands like grep, cat, and ls. Since agents have already learned from vast amounts of Unix interaction data, they can use this immediately without any separate tool training.

nia-docs is a tool that, given a specific URL, crawls the site and mounts it as a file system.

• Indexing: It complies with llms.txt, detects OpenAPI specs, and normalizes complex URL structures into intuitive file paths.
• Serving: It provides operations such as file reading, directory listing, and regex search (grep) through API endpoints, with all responses gzip-compressed for efficiency.

This approach is more general-purpose than MCP, which requires defining a JSON schema every time, and helps agents work while grasping the full context of the documentation.