AI Briefing#data-engineering
The latest AI and developer news about #data-engineering, with the original source and a short summary.
The latest AI and developer news about #data-engineering, with the original source and a short summary.
An Olive Young data engineer shares news on AI agents and new hardware experienced on-site at Google Cloud Next '24.
Wanted Lab's Data Team translated the impact of its Weaver project into time savings, replacement costs, and infrastructure savings.
Banksalad developed 'dataserving,' a platform that converts analytics data into production APIs, improving the development efficiency of data products.
Banksalad has introduced 'datapipe,' an internal tool that lets data analysts directly develop and operate pipelines, resolving the engineer bottleneck.
Engineers from Kakao's Hadoop Engineering Team held a small talk for 42 Seoul cadets, sharing insights on data roles and required competencies.
Olive Young is building out its data ecosystem, expanding from an existing IDC-based ETL structure to a cloud-based data platform leveraging GCP.
We operate an internal SaaS platform that supports the entire process from data collection to ML/AI utilization for Kakao crew members.
The team productizes everything from user profiling to search and recommendation modules, breaking down role boundaries between analysts and engineers.
It performs 190,000 analysis jobs daily across 140 shared Hadoop clusters and 9,100 nodes.
Kurly data analysts ensure data quality through a 6-step process ranging from requirement refinement to production/analytics separation and consistency validation.
With AQE enabled, the number of Shuffle partitions decreased from 10000 to 118, and Peak Memory dropped from 634.8GB to 332.0GB.
Six technical sessions covering real-time classification, Airflow 2.0, PostgreSQL, and more will be presented on Day2.
Kakao Data PE Cell shared a case study where query improvements and partition count adjustments reduced execution time from 8.4 hours to 3.5 hours through Spark Shuffle Partition optimization experiments.
Banksalad has built a Data Discovery Platform that leverages DataHub to simplify the process of searching for and understanding data.
This covers how to build an efficient testing environment for a data analytics library that uses PySpark and S3.
Kakao data engineers handle everything from pipeline construction to collaboration with data scientists.
This introduces the background and problems that led Devsisters to develop the **DevPlay Analytics** library to improve the convenience of using PySpark SQL.
Devsisters' Data Platform Cell uses engineering to build an environment where data flows smoothly and members can make data-driven decisions.