Hugging Face Optimizes Parquet Deduplication

Hugging Face currently hosts approximately 11PB of datasets, of which 2.2PB consists of Parquet files, making storage optimization a critical challenge.

The existing Byte-level Content-Defined Chunking (CDC) approach shows a high deduplication rate of 99.1% for data Appends, but efficiency drops sharply for data Modifications or Deletions.

• Modification: Since Parquet column headers contain absolute file offsets, even small modifications cause the header to be rewritten, dropping the deduplication rate to 89%.
• Deletion: Row Group structures get reorganized, and due to the compression method, data blocks end up being completely regenerated.

To address this, the Hugging Face team is reviewing Content Defined Row Groups as a solution, which divides row groups based on the hash value of a specific key column rather than a fixed number of rows.