Query Dependent Chunking

Existing RAG systems use a single, predetermined chunk size. Small chunks preserve details well but lack context, while large chunks provide rich context but dilute specific facts—creating a trade-off.

Research has revealed that even within the same corpus, the optimal chunk size varies depending on the nature of the query. In other words, there is no single 'sweet spot' that works for all queries.

To address this, the approach proposes indexing the same data using multiple chunk sizes (e.g., 100, 200, 500 tokens) and combining the results via RRF (Reciprocal Rank Fusion). This method is efficient since it requires no model retraining or use of an LLM to generate separate context.

Experimental results show this approach improved retrieval performance by 1-37% across various benchmarks. Additionally, oracle experiments confirmed that selecting the optimal chunk size per query leaves room for an additional 20-40% performance improvement.