Full-Text Search in SmithDB: Inverted Index Design for Object Storage

SmithDB supports Full-text search and JSON filtering for agent traces with a median latency of 400ms (P50), even in environments where large, nested JSON documents are stored in object storage.

Agent traces have unique characteristics that differ from traditional log data. The inputs and outputs fields account for the majority of the total data, and as LLM context windows grow larger, payload sizes tend to expand to hundreds of MB. This results in a source:index ratio of 1:1.9, much higher than typical logs (1:1.25).

SmithDB applied the following design principles to address these challenges:

• Efficient index structure: To handle token frequencies that follow a Zipfian distribution, SmithDB adopted a structure that can compactly store terms of varying frequencies within a single file.
• Object storage optimization: Considering per-request latency and throughput limits, the storage layout and query execution were designed to minimize unnecessary data reads.
• Support for diverse query modalities: SmithDB supports path existence checks for verifying whether a specific path exists, keyed value searches for finding specific values, and free text search for searching free-form text.