HuggingFace Improves Data Streaming by 100x

HuggingFace has significantly improved the streaming (streaming=True) feature of the datasets library, making it much faster and more stable to train on terabyte (TB)-scale large datasets without downloading them.

The key improvements are as follows:

1. Startup Optimization

• Data File Caching: Instead of the previous approach where every DataLoader worker sent individual requests, only the first worker fetches the file list while the rest read it from a local cache, resolving the request storm problem.
• Resolution Logic Optimization: Minimized the number of API calls, cutting the time needed to check the data file list by 10x.

2. Streaming Performance Improvements

• Parquet Prefetching: While the model processes the current data, the next data chunk is fetched in the background in advance, minimizing GPU wait time.
• Configurable Buffering: Provides options for users to directly adjust the buffer block size and prefetch amount according to their hardware and network environment.

With this update, the number of startup requests decreased by up to 100x, data resolution speed became 10x faster, and streaming speed became up to 2x faster.