xerj: Cut tokens by 2.7x with a local search engine that replaces grep

A local search engine that reduces the tokens an AI agent consumes by reading files one by one. It indexes various formats such as code, documents, logs, and PDFs as a single Rust binary, supporting BM25, vector, and hybrid search. It provides an Elasticsearch-compatible API for immediate integration with existing clients.

Instead of putting entire files into the context, the agent receives only the necessary paths, improving work efficiency. Measurements show that it reduces output tokens by 26x compared to memory-dependent methods and by 2.7x compared to grep-based methods while maintaining the same problem-solving rate. It is particularly effective for internal code or the latest libraries that the model has not been trained on.

It operates as an MCP server, connecting to various AI coding tools such as Claude Code and Cursor. It automates a 'reference coding' workflow where the agent indexes similar open-source projects and searches for existing solutions to write code. It also allows managing agent memory across sessions through the /_memory namespace.

Distributed under the Apache-2.0 license, it keeps data local to maintain privacy. Without complex schema configuration or pipeline setup, simply specifying a folder automatically detects file types and starts indexing. It is also useful for large-scale searches of specific patterns, such as security audits or log analysis.