AI Briefing#apache-spark
The latest AI and developer news about #apache-spark, with the original source and a short summary.
The latest AI and developer news about #apache-spark, with the original source and a short summary.
The team fused metrics computation with Ray inference to eliminate data serialization overhead, reducing end-to-end wall-clock time by 25–32% across dataset sizes.
The engineer reduced multi-hour sequential debugging cycles to parallel runs by isolating experiments via Git branches and automating deployment with AI agents.
The system uses a 1:1 mapping between internal Data Projects and dedicated AWS IAM roles to bridge provider and internal identity systems.
AWS Glue 6.0 has been released with a 30% price reduction and full support for Apache Iceberg v3.
WPP integrated its data and platform infrastructure with Google Cloud to reduce AI marketing campaign deployment time from months to days.
Toss Securities shares the technical challenges and solutions for operating Spark Connect as a stable service in a Kubernetes environment.
Google Cloud unveiled new features for Managed Spark clusters, including Lightning Engine, which maximizes performance, and Flexible VM, which increases resource availability.
Snap adopted NVIDIA cuDF to speed up data processing for Apache Spark-based A/B testing by 4x and cut costs by 76%.
Pinterest reduced Spark OOM failures by 96% with Auto Memory Retries.
Databricks and Hugging Face have launched a feature that directly converts Spark dataframes into Hugging Face datasets.
Replit built its own native search engine that quickly finds content among over 100 million Repls and code, and has rolled it out to all users.
With AQE enabled, the number of Shuffle partitions decreased from 10000 to 118, and Peak Memory dropped from 634.8GB to 332.0GB.
Kakao Data PE Cell shared a case study where query improvements and partition count adjustments reduced execution time from 8.4 hours to 3.5 hours through Spark Shuffle Partition optimization experiments.
This introduces the background and problems that led Devsisters to develop the **DevPlay Analytics** library to improve the convenience of using PySpark SQL.
Kakao revealed the architecture and operational technologies of its real-time recommendation system 'Rubix', highlighting 99.998% availability and the ability to handle 200,000 QPS during peak times.