Proposed Method for Comparing Embedding Models

Since the vector spaces themselves are difficult to compare directly when swapping embedding models, a method for comparing the similarity space has been proposed.

Synthetic Query Probing involves creating content pairs such as synthetic questions and document chunks, and comparing the similarity scores generated by multiple embedding models.

• Titan-series models showed a relatively strong correlation in similarity scores despite differences in dimensionality.
• The relationship between scores from Titan and Ada models is non-linear, and their score ranges also differ.
• This approach can help adjust search thresholds when replacing embedding models and compare search results across different models.

The related research is a paper by Marcin Rozmus and Peter van der Putten, scheduled to be presented at Discovery Science 2026.