Hugging Face Improves Data Transfer Speed by 2-3x

Hugging Face introduced a new data management approach based on Content-defined chunking (CDC) technology to optimize upload and download performance for large-scale models and datasets.

The existing file-centric approach required individually processing hundreds of millions of chunks when managing large repositories, causing problems such as a surge in network requests and increased infrastructure costs. To address this, Hugging Face adopted an Aggregation strategy.

• Blocks: Deduplicated data is bundled into blocks of up to 64MB in size. This reduces the number of entries in the content-addressable storage (CAS) by approximately 1,000x.
• Shards: Manages the mapping between files and chunks to identify only the parts that have changed. Chunks that already exist are skipped during transfer, reducing unnecessary network traffic.

Through these optimizations, AI builders can experience 2-3x faster data transfer speeds in some cases. The core underlying technologies, xet-core and hf_xet, have been open-sourced.