How SmithDB Builds Inverted Indexes for Full-Text Search

SmithDB supports full-text search by designing an Inverted Index that uses object storage as its backend. Index creation happens in real time during the ingestion stage, and the most recent data is read directly from local storage to speed up processing.

To efficiently handle large-scale JSON payloads, SmithDB introduced the JSON Tape format. Inspired by Apache Arrow's arrow-json and simdjson, this approach flattens data into (path, leaf_value) pairs through a single-pass iterator, without separate field allocation or number conversion.

To maximize index-building performance, the following optimization techniques were applied.

• String Interning: Maps duplicate JSON paths and token values to unique integer IDs, reducing string comparison costs and cutting build time by about 2.2x.
• Radix Sort: Groups postings with $O(n)$ complexity before passing sorted runs to the FST (Finite State Transducers) writer.
• Flush Thresholds: To prevent tokens of a certain frequency from growing indefinitely, data is flushed based on three thresholds: Row Group (32MB), Aligned Chunk (2MB), and Mid-term Position Spill (8MB).

Through this architecture, SmithDB ensures efficient query performance in an object storage environment while keeping memory usage under control.