AI Briefing#data-pipeline
The latest AI and developer news about #data-pipeline, with the original source and a short summary.
The latest AI and developer news about #data-pipeline, with the original source and a short summary.
As the autonomous driving competition shifts from model performance to the speed of data relearning, AWS and NVIDIA have presented an 8-stage pipeline to address this.
The feature is available in public preview and free to get started.
Recent pipeline changes reduced backbone traffic by ~70 GB/s, saving significant CPU and memory resources.
Transformers are now generally available on all Cloudflare self-serve plans, allowing teams to filter and reshape logs before export using SQL.
Migrated over 1,200 pipelines from Oozie and Sqoop to Airflow and Spark, reducing ingestion speed by up to 70% and improving query performance by 66x.
Yeogi-eottae Company designed a data pipeline to build an Agentic RAG MCP that leverages internal knowledge.
NAVER has unveiled 'Sommelier,' a data pipeline designed to minimize latency in real-time voice conversation AI.
It's a high-performance Rust library that determines whether a PDF contains text without OCR and converts it to Markdown.
Photoroom has revealed the data pipeline and captioning philosophy used to train the PRX model.
Oliveyoung redesigned its pipeline with an event-driven architecture (EDA) to resolve the delays and inefficiencies in inventory data processing that arose during omnichannel expansion.
India is rapidly emerging as a global hub for collecting egocentric (first-person view) data, which is essential for robot learning.
Pinterest built a next-generation DB Ingestion Framework that automatically handles schema changes to improve the efficiency of its data pipelines.
Netflix built a real-time traffic-based 'Data Canary' system to validate the integrity of data pipelines, not just code deployments.
The Duckle MCP server has been released, allowing Claude to build data pipelines directly in a local environment.
By tying together product, coupon, gift, and promotion benefit data through Kafka, delays of up to 45 minutes were cut down to a matter of seconds.
50+ AI agents organized the tribal knowledge of 4,100 files into 59 context files.
To solve the data latency problem of its existing ODI-based batch system, Oliveyoung built a hybrid CDC architecture leveraging OGG and Kafka.
Replaced the existing complex batch-based synchronization structure with Debezium and Strimzi-based Kafka Connect to resolve consistency issues.
AI handles primary filtering to reduce operators' psychological burden and improve judgment consistency.
Kakao has revealed a Spark job design featuring four-stage modularization and design patterns, including Reader and Translator, to ensure data reliability across approximately 300 CDC pipelines.
Kakao shares its Spark job development case study, applying MySQL scan mode tuning, Iceberg maintenance, caching, and local mode optimization to handle large-scale consistency checks across hundreds of tables.
This proposes a method for building a data pipeline that minimizes padding waste to maximize multimodal model training efficiency.
Devsisters shared the details of its 2nd Engineering Day - Data event held on May 29, 2025.
Kurly shared the data loss limitations of the Kafka Connect JDBC source connector and how to use the timestamp.delay option.