AI Briefing#kafka
The latest AI and developer news about #kafka, with the original source and a short summary.
The latest AI and developer news about #kafka, with the original source and a short summary.
To solve the data latency problem of its existing ODI-based batch system, Oliveyoung built a hybrid CDC architecture leveraging OGG and Kafka.
They used CloudWatch and Redis Stream to detect DynamoDB load and pause/resume Kafka consumption.
Stabilized spike settlement using Kafka Streams windows and a timestamp extractor.
We internalized an external Batch Scheduler using Quartz and a common library, calling it OnQuartz.
Lotte ON internalized a real-time image processing service using S3 Object Lambda and CloudFront.
Olive Young participated in SpringCamp 2025 as a Gold sponsor, presenting logistics system improvement case studies and engaging with developers.
This covers the technical challenges and solutions Olive Young faced while building its own Kafka-based global logistics system (GMS) without relying on external solutions.
By replacing Spring Batch synchronous calls with coroutine-based asynchrony, the bottleneck from 15 APIs was reduced.
Oliveyoung introduced a Kafka-based asynchronous request-response architecture to resolve performance degradation issues in its existing synchronous order system.
Using Flink's KeyedProcessFunction, they implemented real-time event combination and zero-downtime deployment.
The team indexed applicable products for coupons into Elasticsearch, reflecting coupon and product changes in real time.
In server log Iceberg APPEND loading tests, applying zstd level 9 reduced file size by 51.4% compared to ORC.
Tracks interface history and call stacks using AOP, MDC, and Kafka Interceptor.
The S3+GCS batch pipeline was transformed into a near-real-time structure based on MSK and Protobuf.
They replaced a legacy Flink app running on 96 CPUs with Flink SQL and GitOps deployment.
Proxies, rollback marking, propagation options, and Kotlin exception conversion determine what actually gets rolled back.
KakaoPay's Information Platform Team shares their case and benefits of adopting Spring Cloud Stream for data composition and pipeline processing.
Kakao Pay redesigned its RabbitMQ-based delayed transfer service using Kafka, improving processing speed by 8x.
Achieved a 50% reduction in server resource usage and a significant decrease in memory usage from 90% to 15%
Kellyz's Order Platform Development team combined Spring Kafka's ConsumerSeekAware with Redis Pub/Sub to shift offsets without stopping the consumer group.
Devsisters built a near real-time metrics pipeline guaranteeing under 10 minutes of latency to enable rapid decision-making during game launches.
Shares how to maintain data consistency when failures occur while operating Debezium MSK Connect, using Offsets and creating new Connectors.
Kelly completed a full switch from its SaaS-based delivery system to an in-house developed system, achieving zero failures during the transition.
SK Planet developed 'Router', a real-time data pipeline service combining Kafka and Spark Streaming, to solve the latency issues of existing batch processing.