Material Reconstruction and Blending in CLIP Space

MARBLE is a method for editing an object's material based on an example material from an image. By finding material embeddings in CLIP-space and using them to control a pretrained text-to-image model, it handles material blending and reconstruction simultaneously.

The key lies in identifying, within the denoising UNet, the blocks responsible for material properties, allowing the material information from the example image to be reflected more accurately. Using the representation obtained this way, a direction between two material example images can be found in CLIP-space to perform blending.

In addition, fine-grained material attributes such as roughness, metallic, transparency, and glow can be parametrically controlled by predicting a direction of change with a shallow network. This allows multiple edits to be performed simultaneously in a single forward pass, and it can also be applied to painting tasks.

Qualitative and quantitative analyses also demonstrated the effectiveness of the proposed method, suggesting a direction toward making example-based material editing more fine-grained and flexible.