Hugging Face Launches Storage Buckets for ML Artifacts

Hugging Face has introduced Storage Buckets for managing frequently generated intermediate ML artifacts such as checkpoints, optimizer states, and data processing shards. Unlike the existing Git-based repositories, it provides S3-style object storage that allows large volumes of data not requiring version control to be written and overwritten quickly.

Key Features:

• Xet-based deduplication: Using Xet, a chunk-based storage backend, when storing files with similar content (e.g., consecutive checkpoints), duplicate data is excluded from transfer, significantly reducing bandwidth and storage costs.
• Pre-warming feature: Data can be pre-fetched to a specific region of a cloud provider (AWS, GCP, etc.), minimizing data transfer latency during distributed training.
• Easy management: Bucket creation, data synchronization (sync), and listing can be performed instantly from the command line via the hf CLI, and browsing is also available on the Hub webpage.

This feature is optimized for engineers operating large-scale training clusters or data pipelines to efficiently manage the massive intermediate outputs generated during the model training process.