AI Briefing#spark
The latest AI and developer news about #spark, with the original source and a short summary.
The latest AI and developer news about #spark, with the original source and a short summary.
The system wraps deterministic processes in AI to extract errors and provide focused diagnosis, replacing legacy regex rules that left 50% of failures unclassified.
Migrated over 1,200 pipelines from Oozie and Sqoop to Airflow and Spark, reducing ingestion speed by up to 70% and improving query performance by 66x.
Google is adding image and video generation and camera features, among others, to the Gemini desktop app.
Pinterest built a unified CDC-based DB ingestion framework to reduce data latency and cut costs.
By turning code-managed user behavior logs into a UI-based platform, Karrot improved consistency and automation.
Dropbox built a feature store for sub-100ms ranking using Feast, Go, and Dynovault.
Kakao has revealed a Spark job design featuring four-stage modularization and design patterns, including Reader and Translator, to ensure data reliability across approximately 300 CDC pipelines.
Kakao shares its Spark job development case study, applying MySQL scan mode tuning, Iceberg maintenance, caching, and local mode optimization to handle large-scale consistency checks across hundreds of tables.
Banksalad solved a surge in Athena costs when querying S3 data by optimizing with Bucketing and Partition structure, reducing costs by about 700x.
Banksalad switched its Spark computing environment from EMR to Kubernetes, reducing operating costs by about 45%.
Olive Young is building out its data ecosystem, expanding from an existing IDC-based ETL structure to a cloud-based data platform leveraging GCP.
The team productizes everything from user profiling to search and recommendation modules, breaking down role boundaries between analysts and engineers.
With AQE enabled, the number of Shuffle partitions decreased from 10000 to 118, and Peak Memory dropped from 634.8GB to 332.0GB.
Kakao Data PE Cell shared a case study where query improvements and partition count adjustments reduced execution time from 8.4 hours to 3.5 hours through Spark Shuffle Partition optimization experiments.
Devsisters' Data Platform Cell uses engineering to build an environment where data flows smoothly and members can make data-driven decisions.