VDR Performance Reversal

This explains how to extend Sentence Transformers' multimodal training pipeline in the same way as for text, covering the flow of directly finetuning embedding and reranker models that handle images, video, and audio.

The key example is a case of tuning Qwen/Qwen3-VL-Embedding-2B for Visual Document Retrieval (VDR). Using 53,512 English samples from a public document-query dataset, training with a query-correct image-hard negative structure raised NDCG@10 from 0.888 → 0.947. The author states that this result outperformed all existing multimodal VDR models tested, and in some cases even surpassed models 4x larger.

The training setup is organized the same way as for text models:

• Model: an existing multimodal embedding model or a VLM checkpoint
• Dataset: input and evaluation sets containing images
• Loss Function: a loss for retrieval
• Training Arguments: batch size, precision, logging/save intervals
• Evaluator / Trainer: for running evaluation and training

On the model side, two approaches are presented:

• Loading an existing multimodal embedding model like SentenceTransformer("Qwen/Qwen3-VL-Embedding-2B"), and adjusting resolution, bfloat16, flash_attention_2, etc. via processor_kwargs and model_kwargs
• Starting from a new VLM checkpoint like SentenceTransformer("Qwen/Qwen3-VL-2B") and letting Sentence Transformers automatically recognize the modality

Another method introduced is using Router to separate the text encoder and image encoder, then aligning them into a shared embedding space with Dense projection. This approach is useful when combining lightweight, specialized encoders, but requires training to align the spaces.

The data used is tomaarsen/llamaindex-vdr-en-train-preprocessed. The original is a multilingual query-image dataset of roughly 500,000 samples, and this article refines the English subset into a form ready for immediate training. The training input uses query, image, and negative_0 to construct (anchor, positive, hard negative) triplets.

For the loss function, CachedMultipleNegativesRankingLoss is used to:

• utilize explicit hard negatives, and
• reuse other samples' positives/negatives within the batch as in-batch negatives

This is wrapped with MatryoshkaLoss to train so that performance is maintained across multiple dimensions such as 2048/1536/1024/512/256/128/64. This allows the embedding dimension to be truncated at deployment time to balance search speed and quality.

In the training configuration, per_device_train_batch_size=64, bf16=True, and BatchSamplers.NO_DUPLICATES are used to efficiently train the large VLM, while mini_batch_size=1 reduces memory load. It also emphasizes specifying eval_steps, save_steps, and logging_steps as fractions of an epoch to periodically check progress.