Hugging Face Acquires XetHub

Hugging Face announced that it has acquired XetHub for the efficient management of large-scale AI datasets and models. XetHub is a company with technology that enables Git to support large repositories at the terabyte (TB) scale, founded by a team that built Apple's ML infrastructure.

Through this acquisition, Hugging Face plans to build its own optimized storage and version control backend to replace the existing Git LFS. This will provide the following technical benefits:

• Chunk-based uploads and deduplication: When only part of a 10GB file is modified, only the changed portion is uploaded instead of re-uploading the entire file, maximizing efficiency.
• Large-scale model optimization: Even when editing metadata in a GGUF model, only small chunks on the order of a few KB need to be updated.
• Enhanced scalability: Building infrastructure that can reliably accommodate future trillion-parameter models and massive datasets.

The Hugging Face Hub currently stores approximately 12PB of data, and this technology integration is expected to provide a better collaboration environment for community and enterprise users working with large-scale models and datasets.