AI Briefing#big-data
The latest AI and developer news about #big-data, with the original source and a short summary.
The latest AI and developer news about #big-data, with the original source and a short summary.
This covers the operational and design challenges LY Corporation faced while integrating 1EB-scale HDFS clusters with different architectures.
Google unveiled innovative technologies behind Dataflow that enhance scalability and efficiency for large-scale machine learning data processing.
Piqama unifies quota lifecycle and right-sizing across Pinterest into a single platform.
KakaoPay shares the cluster configuration and incident response know-how it gained while operating **Apache Pinot**, a real-time OLAP solution.
SK Planet adopted the open-source Trino as its company-wide data analytics engine, resolving data ingestion delays and building a real-time analytics environment.
Kakaopay resolved large-scale data processing issues on its MyData platform through distributed data storage and the adoption of Palsonic.
Banksalad solved a surge in Athena costs when querying S3 data by optimizing with Bucketing and Partition structure, reducing costs by about 700x.
Developed by 3 people over 6 months, the system successfully migrated 4,000 Flows and 40PB of data.
It is being developed and operated in-house with five components to resolve issues of external solution costs and functional limitations.
Kakao's Ad Development Team has opened positions for AdTech platform development roles, including SSP and DSP.
It performs 190,000 analysis jobs daily across 140 shared Hadoop clusters and 9,100 nodes.
With AQE enabled, the number of Shuffle partitions decreased from 10000 to 118, and Peak Memory dropped from 634.8GB to 332.0GB.
Kakao Data PE Cell shared a case study where query improvements and partition count adjustments reduced execution time from 8.4 hours to 3.5 hours through Spark Shuffle Partition optimization experiments.
Kakao data engineers handle everything from pipeline construction to collaboration with data scientists.
Kakao has released 'hbase-tools', an open-source tool that automates metric monitoring, region management, and snapshot management for HBase clusters.
Kakao has released 'HBase Region Inspector', an open-source tool that visualizes region distribution and request status in HBase clusters.