AI Briefing#airflow
The latest AI and developer news about #airflow, with the original source and a short summary.
The latest AI and developer news about #airflow, with the original source and a short summary.
The platform reduces metric creation time from weeks to 1–2 hours and supports over 98% of experimentation metrics.
Migrated over 1,200 pipelines from Oozie and Sqoop to Airflow and Spark, reducing ingestion speed by up to 70% and improving query performance by 66x.
36 open-source Agent Skills for Claude that automatically load data engineering context have been released.
Automatically provisions an isolated Airflow environment within 5 minutes upon PR creation, supporting Jupyter and VS Code for Web.
Apache Flink and RocksDB tuning enabled real-time ad impression aggregation from 1 minute to 7 days.
The S3+GCS batch pipeline was transformed into a near-real-time structure based on MSK and Protobuf.
KakaoBank adopted Airflow to perform efficient fund batch processing while complying with financial regulations in a cloud-native environment.
FAST automatically generates Airflow DAGs through a web UI, making it easy for non-developers to create batches.
Kurly migrated its Airflow operating environment to Kubernetes, cutting costs by 50% and resolving GitSync and OOM issues.
By bundling per-Task failure emails in Airflow into a single message, alert noise was reduced.
By separating tagging logic from storage and adding a write buffer, updates were reduced by 67%.
Banksalad has introduced 'datapipe,' an internal tool that lets data analysts directly develop and operate pipelines, resolving the engineer bottleneck.
Banksalad switched its Spark computing environment from EMR to Kubernetes, reducing operating costs by about 45%.
Coupang built a low-code, no-code workflow that automates data building and service development.
Olive Young is building out its data ecosystem, expanding from an existing IDC-based ETL structure to a cloud-based data platform leveraging GCP.
The team productizes everything from user profiling to search and recommendation modules, breaking down role boundaries between analysts and engineers.
Six technical sessions covering real-time classification, Airflow 2.0, PostgreSQL, and more will be presented on Day2.
Banksalad has built a Data Discovery Platform that leverages DataHub to simplify the process of searching for and understanding data.