Manticore Search Introduces Internal Chunking to Improve Long Document Search Performance

Manticore Search has introduced the chunk_strategy feature to prevent information loss when embedding long documents that exceed the model input window (e.g., 512 tokens). This enables chunking and automatic embedding to be performed internally within the engine, without requiring a separate pipeline at the application level.

Chunking Strategies and Performance Improvements

Among the five provided strategies (truncate, mean, fixed, recursive, sentence), the recursive strategy significantly improves deep content search performance by splitting documents at natural boundaries. Benchmark results on the Manticore manual show that recall@5 for content outside the model window improved from 55.1% to 83.3%, and MRR improved from 0.44 to 0.70. Additionally, top-1 accuracy (hit@1) increased from 33.7% to 58.7%.

Resource Costs and Tuning

Performance improvements come with resource costs. Index RAM increases by approximately 2.5x (4.2MB → 11.7MB), the number of vectors increases by approximately 25x (189 → 4,664), and data loading time increases by approximately 4x (21s → 86s). Query p50 response time also increased from 6.3ms to 8.5ms. A balance between performance and resource usage can be achieved through parameters such as chunk size (max_tokens), overlapping tokens (overlap_tokens), and maximum number of chunks (max_chunks).

Implementation Features and Limitations

• Multi-vector Processing: The recursive, fixed, and sentence strategies generate multiple vectors in the float_vector_array column. During search, documents are returned as a single result, and the distance of the closest chunk is reported.
• Comparison with Existing Systems: While most vector databases like Milvus and Qdrant require application-level chunking, Manticore and Elasticsearch support field-level chunking. Manticore offers a simpler implementation with just one column option.
• Caution: If max_chunks is exceeded, merging and truncation may occur in the last chunk without warning.