AI Briefing#kafka
The latest AI and developer news about #kafka, with the original source and a short summary.
The latest AI and developer news about #kafka, with the original source and a short summary.
Eleven technical sessions, including KakaoPay's distributed lock for payments and KakaoWork's Rust migration of its messaging engine, will be held on October 24.
Nine sessions covering MongoDB auto-upgrades, Kafka Kraft operations, and large-scale data synchronization were conducted.
Kakao revealed a total of 29 technical sessions across four fields—AI, Backend, Cloud, and Data—for the third day of if(kakaoAI)2024.
This shares a case from Oliveyoung's OMS project where Kafka was used to prevent message duplication and loss while securing high availability.
The integration with MySQL based on Flink CDC 2.4.1 applied step-wise Kafka topic separation and DDL security response logic.
To resolve delays in session_id ingestion caused by traffic spikes, the team separated it out using Redis Stream.
Migrating Deal lookups from Oracle to MongoDB improved performance and operability.
Kakaopay Securities built a CDC pipeline using Kafka and Debezium to transfer data in real time from Oracle to MongoDB.
By introducing Spring Batch and Kafka, the team resolved server load and user wait-time issues that occurred during large-volume data extraction.
Oliveyoung's external seller service introduced an architecture that protects internal systems from spiky traffic through asynchronous processing and service separation.
Oliveyoung's Retail Platform Development Team shares how it has run its team study for one year and the growth experience gained along the way.
By separating tagging logic from storage and adding a write buffer, updates were reduced by 67%.
KakaoBank introduced ML models and a real-time pipeline to overcome the limitations of its existing Rule-based FDS.
Oliveyoung built a new inventory system leveraging Redisson and Kafka to distribute DB load and transition to MSA.
Drawing on the Ad Recommendation Team's experience with real-time data processing, the session introduced case studies on applying reliability options such as Exactly-Once.
Olive Young's Coupon Squad adopted RabbitMQ to overcome the limitations of their existing Redis Worker, securing system stability and scalability.
Kurly used TestContainers to automate integration tests that closely resemble real environments, improving deployment stability.
Kurly reduced data loading time from 30 minutes to 13 seconds and improved cost efficiency by adopting BigQuery.
Kurly resolved the latency and storage issues of its existing data warehouse by adopting BigQuery.
A custom connector developed for ad log filtering and branching by placement, supporting JSON filtering and sampling.
Developed by 3 people over 6 months, the system successfully migrated 4,000 Flows and 40PB of data.
Built a system using Druid and Kafka to collect and visualize metrics and logs in real time across more than 100 clusters.
A junior developer at KakaoPay shares their experience of discovering a log pollution issue during MongoDB connection pool configuration and contributing to open source.
Kurly applied a Kafka Batch Listener to prevent rebalancing and significantly increase search indexing processing speed.