Hugging Face Reveals Hybrid Search Architecture for Papers with Code

Hugging Face has built a PostgreSQL-based hybrid search system targeting over 110,000 papers on Papers with Code. This system enhances search accuracy by combining Full-Text Search, which enables precise keyword matching, and vector search (pgvector), which identifies semantic similarity, using the Reciprocal Rank Fusion (RRF) algorithm.

Architecture and Tech Stack

The system utilizes Hugging Face Jobs, Storage Buckets, and Inference Endpoints to separate offline embedding generation from online search. The embedding model used is Qwen3-Embedding-0.6B, which applies Matryoshka Representation Learning (MRL) to reduce the dimensionality from 1024 to 256, cutting storage space by approximately 27% while maintaining high recall. Query embeddings are processed in real-time via low-latency Inference Endpoints, with a fallback mechanism that responds immediately using only Full-Text Search in case of failure.

Performance and Operational Strategy

In pilot tests (based on 5,000 papers), pgvector search achieved Recall@20 0.9955 and a p95 latency of 2.21ms. When combined with RRF, equal weights are applied, using the k=60 parameter. Operationally, the system fixes model revisions and dimensions, ensures data immutability through checksums, and supports safe deployment via atomic activation. This enables researchers to find accurate papers even with typos or incomplete titles.