Apple Researchers Propose Amortized MIPS to Speed Up RAG Retrieval with Learned Neural Networks

Apple researchers introduced Amortized MIPS, a method that trains small neural networks offline to predict Maximum Inner Product Search (MIPS) solutions, improving both speed and accuracy for Retrieval-Augmented Generation (RAG) systems.
On the Natural Questions dataset (2.6M passages), the KeyNet model reduced latency by up to 8x (0.23 ms vs 1.8 ms) and required up to 11x fewer key scans (0.009% vs 0.10%) to achieve 99% recall compared to original queries.
The SupportNet model improved routing efficiency, achieving 95% recall 1.9x faster than centroid routing (1.74 ms vs 3.36 ms) and identifying 90-93% of true partitions when scanning only 5% of the database.
The approach encodes MIPS solutions into network weights, amortizing the cost of repeated search operations into a one-time training phase, making it suitable for production systems with known query distributions.

Model Architectures

SupportNet uses Input Convex Neural Networks (ICNN) to learn the support function directly, providing calibrated scores for re-ranking and thresholding.
KeyNet acts as a query rewriter, outputting the optimal matching key vector to guide existing Inverted File Index (IVF) systems toward faster searches.

KeyNet is recommended when inference speed is the bottleneck and only an approximation of the best key is needed.
SupportNet is preferred when calibrated scores are required for downstream tasks or when compute resources are limited for small models.
For large-scale systems, a hybrid approach is advised: using SupportNet for cluster-level routing and KeyNet for intra-cluster search optimization.