Luce: Relightable Gaussians for 3D Asset Generation

Generating 3D assets from high-resolution images requires a representation that captures both geometry and appearance. To support relighting and integration into standard rendering pipelines, it must include physically based rendering (PBR) modalities such as albedo, metal-roughness, and surface normals.

Luce proposes a 3D representation that integrates geometry and PBR materials within a voxelized multimodal Gaussian cloud to meet these requirements. It uses dedicated Gaussian primitives for each modality and compresses this representation into a material-aware latent space via a variational autoencoder (VAE).

A Rectified-flow transformer generates these latent variables from a single image, conditioned on multi-layer features from a pre-trained image encoder. These latent variables are decoded into relightable PBR Gaussians and a textured mesh containing tangent-space normal maps.

Key achievements include:

• Elevating single-image-based 3D generation performance to state-of-the-art on the Toys4K dataset, improving FID by 28% over the strongest baseline.
• Setting a new record on AI-generated image benchmarks with a CLIP image alignment score of 0.8519, surpassing the previous best (0.8299).

Luce can generate geometrically accurate, material-faithful, relightable assets while preserving fine details such as text, logos, and engravings.