Introducing Contextual Retrieval

For AI models to be useful for specific tasks, they need background knowledge, and a representative method for supplementing this is RAG. However, traditional RAG loses context in the process of splitting documents into small pieces, causing problems such as missing semantically relevant chunks or failing to find exact terms and identifiers.

As an alternative, Contextual Retrieval is proposed. By prepending a short explanation generated from the entire document to each chunk to simultaneously build Contextual Embeddings and Contextual BM25, the context needed for retrieval is preserved. For example, before embedding a sentence from an SEC filing, information about "which company's and which quarter's document" is prepended first, so that a sentence like "revenue grew by 3%" doesn't lose its original meaning.

The core flow is as follows.

• Split the document into chunks.
• Use Claude to generate a short contextual description for each chunk.
• Prepend that description to the chunk, then build embeddings and a BM25 index.
• At retrieval time, use both semantic search and exact match search together, then combine the results and feed the top chunks into the model.

This method significantly reduces the weaknesses of traditional RAG. Experimental results show that Contextual Embeddings alone reduced the top-20 chunk retrieval failure rate by 35%, and the combination of Contextual Embeddings + Contextual BM25 lowered it to 49%. Adding reranking on top of this reduces the failure rate by 67%, showing potential for improvements in downstream task performance as well.

On the cost side, Claude's prompt caching is important. Instead of re-inputting the entire document every time, it can be uploaded to the cache once and then referenced per chunk, bringing the cost of generating contextualized chunks down to about $1.02 per million document tokens, based on 800-token chunks and 8k-token documents.

However, there are several things to consider when applying this.

• Chunk boundaries and overlap directly affect retrieval performance.
• The degree of improvement varies by embedding model, and the article states that Gemini Text 004 and Voyage embeddings performed particularly well.
• A customized contextualizer with a domain-specific glossary may be better than a general-purpose prompt.
• How many chunks to include with top-K, and how many candidates to look at in reranking, should be checked directly through eval.

Overall, this approach shows how much "better retrieval input" can determine RAG quality more than "a bigger model." If documents are small enough, a simple method of putting the entire document into the prompt is also possible, but as the knowledge base grows larger, a context-preserving retrieval pipeline becomes key.