Decomposing Self-Attention for Training-Free Diffusion Editing

Semantic editing directions are extracted directly from the self-attention weights of pretrained diffusion models. Without separate training or large-scale sampling, eigen decomposition is used to find latent editing axes, precisely changing facial attributes such as age, hairstyle, and eye size.

The core is the latent whitening assumption that, in the early denoising interval of 0.5T~0.8T, the feature covariance is close to the identity matrix. In this interval, the principal eigenvectors of the combined weight C, which pools the Q/K/V projection matrices, are interpreted as editing directions, and the latent is transformed as Z' = Z + αn to produce the desired change.

• Varying α from -0.4 to 0.4 linearly changes the editing intensity.
• On CelebA-HQ, evaluation was conducted with 15 semantic directions and 100 samples, for a total of 1,500 cases.
• It outperformed existing methods on SSIM, PSNR, FaceNet cosine similarity (ID-Similarity), and Directional CLIP.
• Editing time is around 3 seconds, which is presented as 60% faster than existing methods.
• It was also applied to other pretrained models such as Cars, Cats, and Rooms from LSUN, showing sample-independent general applicability.