AI BriefingHugging Face Adopts Xet-Based Storage
Key point
Hugging Face has introduced Xet storage, which supports byte-level deduplication, to improve the efficiency of large file transfers.
Details
Hugging Face has introduced Xet storage to replace the existing LFS (Large File Storage), and has successfully migrated some model and dataset repositories.
Xet's Core Technology: Content-Defined Chunking (CDC) Existing LFS deduplicates at the file level, so even if only a small part of a large file is modified, the entire file has to be re-uploaded, which is inefficient. Xet, on the other hand, splits data into chunks of about 64KB and performs deduplication at the byte level.
Key Benefits and Performance Improvements
- Efficient updates: When adding a row to a Parquet file or modifying metadata in a GGUF model, only the changed chunks need to be transferred.
- Transfer speed innovation: For example, when adding 1MB to a 5GB SQLite database, LFS has to re-upload the entire 5GB, whereas Xet transfers only the changed portion, dramatically reducing upload time.
Hugging Face is currently routing about 6% of its total download traffic through the Xet infrastructure to verify real-world performance and integration across various libraries and cloud environments.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Google Includes Premium Colab Benefits in AI Subscription TiersPirate Face Launched: Decentralized Infrastructure to Protect Hugging Face Models from Deletion