MultiMatte, a Promptable Background Removal Model Based on SAM 3, Released, Achieving S-measure of 0.901 on DIS-VD Benchmark

Feyn Labs has released the MultiMatte model, which removes backgrounds while keeping only specified objects using text prompts. This model is based on Meta's SAM 3 and applies Low-rank fine-tuning (LoRA), modifying only 2.27% of the total parameters.

Technical Features and Training Method

• Alpha Matte Application: To overcome the limitations of SAM 3's binary masks, it uses alpha mattes with continuous transparency values to more accurately represent fine details such as hair and blurred boundaries.
• Efficient Fine-tuning: Leveraging PEFT and LoRA, pre-trained weights are frozen, and Rank-16 adapters are applied to the attention and MLP projections of all towers. The final adapters are merged into the released weights, requiring no additional libraries for inference.
• Training Data: Trained on 19,953 images including salient objects, camouflage, high-resolution, hair, and marine scenes, with 24.8% of them using human-written labels for prompt supervision on specific object names.

Performance Improvement Results

MultiMatte showed performance improvements over SAM 3 across all 12 benchmark splits. Notable improvements were observed particularly in the Dichotomous segmentation benchmark.

• DIS-VD: S-measure increased from 0.667 to 0.901 (+0.233)
• Salient Objects: Recorded improvements of +0.109 on DUT-OMRON and +0.084 on UHRSD-TE
• Camouflage: Significant performance boost of +0.148 on COD10K-TE
• Prompt Effect: Performance improves even without fine-tuning by simply inputting concept names; applying actual concept names to SAM 3 on DIS-VD increased the S-measure by +0.150.

Implementation and Usage

• Based on NoBg Library: Installable via pip install nobg, and RGBA cutouts can be generated using the code model.predict(processor, "image.jpg", "concept").
• Versatility: Demonstrated strong generalization capabilities by achieving high performance on DAVIS-S and DUT-OMRON benchmarks, which were not included in the training data.