Local RAG Search Tool for 200 Million Web Pages

LLMSearchIndex is an open-source library designed for RAG (Retrieval-Augmented Generation) applications, enabling internet-scale web search to be performed locally without external API calls.

Key Features and Data:

• Massive Dataset: Utilizing Wikipedia and FineWeb data, it indexes approximately 203 million web pages.
• Optimized for Local Execution: No external API calls are needed at query time, which is advantageous for security and speed.
• Efficient Architecture: It uses the all-MiniLM-L6-v2 embedding model, reduces dimensionality from 384 dimensions to 64 dimensions via PCA (Principal Component Analysis), and applies Binary Quantization to maximize search efficiency.
• Search Method: Based on a FAISS index, it supports optional reranking using cosine similarity as needed.

System Requirements:

• RAM: Approximately 6 GB
• Disk Space: Approximately 10 GB
• Hardware: Supports CPU inference, with GPU being optional.