AI Briefing#data-pipeline
The latest AI and developer news about #data-pipeline, with the original source and a short summary.
The latest AI and developer news about #data-pipeline, with the original source and a short summary.
KakaoPay's Information Platform Team shares their case and benefits of adopting Spring Cloud Stream for data composition and pipeline processing.
KakaoBank adopted Airflow to perform efficient fund batch processing while complying with financial regulations in a cloud-native environment.
Devsisters built a near real-time metrics pipeline guaranteeing under 10 minutes of latency to enable rapid decision-making during game launches.
SK Planet developed 'Router', a real-time data pipeline service combining Kafka and Spark Streaming, to solve the latency issues of existing batch processing.
FAST automatically generates Airflow DAGs through a web UI, making it easy for non-developers to create batches.
The integration with MySQL based on Flink CDC 2.4.1 applied step-wise Kafka topic separation and DDL security response logic.
To resolve delays in session_id ingestion caused by traffic spikes, the team separated it out using Redis Stream.
This introduces a case where the Product Squad adopted Debezium and AWS MSK Connect for data synchronization during their MSA transition.
LlamaIndex has unveiled LlamaCloud, a managed RAG pipeline featuring complex PDF parsing.
Oliveyoung's commerce team built a Serverless-based ranking system using AWS Glue to overcome the limitations of the existing Oracle procedure approach.
Banksalad developed 'dataserving,' a platform that converts analytics data into production APIs, improving the development efficiency of data products.
Banksalad has introduced 'datapipe,' an internal tool that lets data analysts directly develop and operate pipelines, resolving the engineer bottleneck.
Kurly reduced data loading time from 30 minutes to 13 seconds and improved cost efficiency by adopting BigQuery.
Kurly resolved the latency and storage issues of its existing data warehouse by adopting BigQuery.
A custom connector developed for ad log filtering and branching by placement, supporting JSON filtering and sampling.
Kakao data engineers handle everything from pipeline construction to collaboration with data scientists.
Kurly's development team transitioned their complex SQL-based legacy order monitoring system to Kafka and the ELK Stack, improving query speed to around 1 second.