Research on Domain Generalization via Grounding Visual Representations Using Text - LG AI Research

Existing machine learning models are vulnerable to the Domain Generalization (DG) problem, where performance drops sharply when the distribution of training data differs from that of test data. To address this, this research proposes a Natural Language Supervision approach that leverages human linguistic descriptions.

The core consists of two modules that help the model grasp the core semantics of an image.

• Visual and Textual Joint Embedder: Aligns the visual representations extracted from images with sentence semantics, enabling the model to learn core features that are invariant to domain shifts.
• Textual Explanation Generator: Generates sentences based on visual cues that explain the model's decision rationale, helping to ground the learning process.

This methodology is used to ground the model during the training phase; it is unnecessary at test time but can optionally be used to explain the decision rationale. Experimental results demonstrated outstanding generalization performance, achieving SOTA (State-of-the-art) performance on both the self-constructed CUB-DG dataset and the DomainBed benchmark.