Multimodal Embeddings and Reranker Models with Sentence Transformers

Sentence Transformers v5.4 fully opens up multimodal input. Now you can embed and compare text, image, audio, and video with the same encode() API, making it possible to build semantic search, RAG, and cross-modal retrieval in a single flow.

A multimodal embedding model places different modalities into the same embedding space, while a reranker (CrossEncoder) directly scores the relevance of query-document pairs. The former is well suited for fast candidate retrieval, the latter is more accurate but slower, and combining the two lets you cleanly implement the retrieve and rerank pattern.

Usage isn't very different from existing Sentence Transformers. For image support, install the sentence-transformers[image] extras; for audio, sentence-transformers[audio]; for video, sentence-transformers[video]. The examples use Qwen/Qwen3-VL-Embedding-2B and Qwen/Qwen3-VL-Reranker-2B. However, since the integration PRs for these models haven't been merged yet, you need to specify revision="refs/pr/23" or revision="refs/pr/11".

The embedding model accepts inputs like URLs, local file paths, and PIL Image objects to create image embeddings, and also computes cross-modal similarity with text. Due to the modality gap, it's normal for scores to come out lower than text-to-text comparisons, and what matters is the relative ranking rather than the absolute value. For retrieval use cases, using encode_query() and encode_document() automatically applies the query/document prompts defined by the model.

The reranker ranks mixed-modality documents more directly. In the example, rank() ranked a car image at #1 among candidates that mixed a car image, a text description, and an image+text document, and supports() and modalities let you check which combinations of inputs the model can handle. When needed, predict() can be used to obtain raw per-pair scores for more precise post-processing.