AI Briefing#parquet
The latest AI and developer news about #parquet, with the original source and a short summary.
The latest AI and developer news about #parquet, with the original source and a short summary.
The design minimizes network round-trips by storing indexes in single Parquet 'superfiles' and using a manifest to eliminate initial metadata fetches.
Migrated over 1,200 pipelines from Oozie and Sqoop to Airflow and Spark, reducing ingestion speed by up to 70% and improving query performance by 66x.
3.4 million panels and 19,000 arrays were analyzed using DuckDB and H3.
GM-SEUS v2 has released 3.42 million panels and a new rooftop array.
Parquet CDC functionality has been added to PyArrow and Pandas, improving data transfer and storage efficiency on the Hugging Face Hub.
In server log Iceberg APPEND loading tests, applying zstd level 9 reduced file size by 51.4% compared to ORC.
Hugging Face analyzed deduplication issues in Parquet files for large-scale dataset efficiency and proposed improvements.
A new feature has been added that allows datasets on Hugging Face Hub to be analyzed directly with SQL via DuckDB.