AI BriefingOne Layer Is Enough: Adapting Pretrained Visual Encoders for Image Generation
Key point
The paper proposes the FAE framework, which uses only a single attention layer to transform pretrained visual representations into a low-dimensional latent space suitable for image generation.
Details
Existing visual generative models operate in a compressed latent space for efficiency, but there is a fundamental mismatch in feature distributions of different dimensions between understanding-oriented pretrained encoders and generation-oriented latent spaces.
To address this, the proposed FAE (Feature Auto-Encoder) uses only a single attention layer to transform pretrained visual representations into a low-dimensional latent space suitable for generation. The core of this framework is combining two separate deep decoders.
- First decoder: trained to reconstruct the original feature space
- Second decoder: takes the reconstructed features as input and generates the image
FAE is compatible with various self-supervised encoders such as DINO and SigLIP, and can be readily applied to diverse generative model families such as diffusion models and normalizing flows.
In terms of performance, on the ImageNet 256×256 benchmark, FAE achieved SOTA-level performance with FID 1.29 when using CFG (Classifier-Free Guidance), and achieved FID 1.48 even without CFG, demonstrating both high quality and fast training speed.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.