Building Hybrid Search on HF Infrastructure for Papers with Code

Papers with Code built a hybrid search system to improve access to open AI research and make SOTA models easier to find. This system combines PostgreSQL's Full-text search (keyword search) and pgvector (vector search) to ensure both accuracy and semantic similarity.

The core architecture separates offline corpus building from online search services.

• Hugging Face Jobs: Provides burstable GPU compute for batch workloads processing a corpus of over 110,000 papers.
• Hugging Face Storage Buckets: Handles persistent data transfer between databases, experiments, and Jobs.
• Hugging Face Inference Endpoints: Provides low-latency inference for real-time query embeddings, with an immediate fallback to full-text search in case of service failure.

The embedding model used is Qwen3-Embedding-0.6B, leveraging MRL (Matryoshka Representation Learning) to optimize storage space with 256-dimensional vectors. Additionally, different prompts are applied to queries and documents respectively to enhance search quality.