Hugging Face Strengthens Large-Scale Dataset Hosting and Exploration Features

Hugging Face Hub provides an environment for reliably hosting and sharing terabyte (TB)-scale large datasets. It supports data streaming through the 🤗 Datasets library, helping users test and prototype with minimal computing resources without downloading the entire dataset. A backend update is underway to raise the current per-file limit from 50GB to 500GB.

The Dataset Viewer feature for data exploration is also powerful. Users can view data directly in the browser and quickly find the data they want through Full Text Search and column-based Sorting. This supports various modalities and file formats, including audio, images, and video.

It also supports integration with major open-source data analysis tools such as Pandas, making it easy to apply to existing workflows, and maximizes data usability and reliability through an SQL console and robust security features.