RAG Multi-Vector Retriever for Tables, Text, and Images

The LangChain team stated that providing integrity-preserving question answering across diverse data types, including images, text, and tables, is a core challenge for RAG. To address this, they released three new cookbooks utilizing a multi-vector retriever for documents with mixed content types.

These cookbooks also demonstrate how to combine multimodal LLMs with multi-vector retrievers to enable image-based RAG. The provided examples include:

• Semi-structured data (tables + text) RAG
• Multimodal (text + tables + images) RAG
• Private multimodal (text + tables + images) RAG

LLMs can acquire new information through weight updates (fine-tuning) and RAG. RAG combines the reasoning capabilities of LLMs with the content of external data sources, offering particular strengths in factual recall for enterprise data.

Existing RAG improvement techniques include summary embeddings, windowing, metadata filtering, embedding fine-tuning, and two-stage RAG. However, RAG for documents containing semi-structured and multimodal data remains a challenging area.