Contextual Retrieval

For AI models to work usefully in specific domains, background knowledge is required, and RAG (Retrieval-Augmented Generation) is widely used for this. However, conventional RAG loses context during the chunking and encoding process, resulting in a limitation where it fails to retrieve the information actually needed.

Contextual Retrieval proposes two sub-technologies to solve this problem: Contextual Embeddings and Contextual BM25. Adopting this approach can reduce retrieval failure rates by 49%, and when combined with Reranking technology, the failure rate can be significantly lowered to 67%.

The optimal strategy varies depending on the size of the knowledge base.

• Under 200K tokens: Instead of RAG, the entire knowledge base is directly included in the prompt. Using Claude's Prompt Caching here can reduce costs by up to 90% and increase speed by more than 2x.
• Large-scale knowledge bases: A hybrid approach combining BM25 (exact keyword matching) and Embedding (semantic similarity search) is used.

The hybrid RAG process is as follows.

1. Split the knowledge base into small units.
2. Generate TF-IDF encoding and semantic embeddings for each chunk.
3. Find exact matches with BM25 and semantically similar items with Embedding.
4. Combine the two results through Rank Fusion technology and remove duplicates to extract the optimal information.