Hugging Face Releases Fine-Tuning Guide for TRELLIS.2 3D Generation Models

• Hugging Face provides a script and container for fine-tuning the TRELLIS.2 3D generation model using LoRA adapters on custom 3D assets.
• Fine-tuning on 188 footwear objects from the Google Scanned Objects dataset improved the F-score from 0.276 to 0.803 and reduced Chamfer distance from 0.174 to 0.033.
• The process involves applying LoRA adapters to the Sparse-structure flow DiT and Shape flow DiT, while freezing the Texture flow DiT to prevent catastrophic forgetting due to missing material maps.
• Training runs on a single A100 80GB GPU via Hugging Face Jobs or locally, taking 7.2 hours for the sparse-structure adapter and 3.6 hours for the shape adapter.

Technical Implementation

• Uses DINOv3-L for image feature extraction and BiRefNet for background removal.
• LoRA adapters use rank 8 for sparse-structure attention/MLP and rank 16 for shape attention layers, totaling approximately 9.3M trainable parameters.
• Training utilizes rectified flow matching loss with an optimizer and bfloat16 autocasting.

Availability and Tools

• Code and preprocessed datasets are available on GitHub and Hugging Face Hub.
• Supports execution via Hugging Face Jobs or local Docker containers.
• Metrics are tracked and visualized using Trackio.