Infino details architecture for running full-text search indexes directly on S3 object storage

Infino, a retrieval engine for AI agents, has detailed its engineering approach to hosting full-text search indexes directly on object storage (S3, GCS, Azure Blob) rather than local disks. The core challenge is that while local SSD random reads are nearly free, object storage random reads incur significant latency (tens of milliseconds) and per-request costs. Consequently, the architecture prioritizes minimizing the number of GET requests over optimizing for CPU or byte transfer volume.

Storage Architecture: Superfiles and Manifests

The system organizes data into superfiles, which are immutable Parquet files containing document rows, full-text indexes, and vector indexes. Unlike Lucene, which splits index components across multiple files, a superfile is a single object. This allows any section of the index to be fetched with a single ranged GET request.

To avoid the overhead of opening these files, Infino uses a manifest that stores the Parquet footer and index headers of every superfile inline. Since the manifest is cached, opening a superfile requires zero GET requests; the first request a query makes is for the actual data it needs. The manifest also holds column statistics (min, max, null counts), allowing queries to skip superfiles entirely if they don't match filter criteria.

The Term Index and Query Optimization

A global term index maps every term to the specific superfiles containing it, along with document counts and BM25 score bounds. This structure allows queries to:

• Skip irrelevant files: A query for "refund" checks the term index and only opens superfiles that actually contain the term.
• Optimize Top-K searches: By visiting superfiles in descending order of score bounds, the engine can prune search paths once the k-th best score is found.
• Answer Count Queries Instantly: Single-term counts can be calculated from the term index's document counts without opening any superfiles. In one test, a count query on a table with 141 superfiles went from opening 141 files to opening none.

Caching and Data Access

Infino employs a lazy loading strategy, fetching only the specific byte ranges needed for postings, document lengths, or positions. To mitigate latency, the system caches superfiles on local disk. When a query first touches a superfile, it fetches necessary ranges from object storage while simultaneously downloading the full file in the background. Once downloaded, the file is memory-mapped, eliminating further GET requests for subsequent queries. The disk cache defaults to 10 GB but is configurable.

Despite these storage-layer innovations, the core search algorithms—including postings lists, BM25 scoring, and block-max pruning—remain consistent with Lucene's foundational ideas.