Hugging Face Switches Repository Backend to Xet

Hugging Face is transitioning the Hub's repository system from the existing Git LFS to Xet, which is optimized for large-scale AI workloads. Currently, more than 500,000 repositories (about 20PB in scale) have already been migrated to Xet, and more than 1 million users are using it.

This migration focused on ensuring backward compatibility without service interruption.

• Git LFS Bridge: Enables users with existing clients to access Xet-based repositories without any separate updates.
• Background migration: Moves data around the clock without service interruption, without affecting users' upload/download operations.
• Xet-dedicated clients: Using hf-xet or the latest version of huggingface_hub allows files to be split into chunks for transfer, maximizing efficiency.

Xet is built on a Content Addressed Store (CAS) and a Rust-based client, providing the scalability needed to keep pace with the rapid growth in the scale of AI models and datasets.