AI Briefing
KO

How to Solve Kafka Message Duplication and Loss Cases

·2024.10.17 03:00

Key point

This shares a case from Oliveyoung's OMS project where Kafka was used to prevent message duplication and loss while securing high availability.

1 / 2

Details

Oliveyoung completely overhauled its existing EAI and Batch-based integration approach into a Kafka-based real-time interface through the OMS (Order Management System) project. Through this, it improved throughput by up to 45x compared to before, building a stable data transmission and reception structure.

To solve the duplication and loss issues that can occur during message transmission, the following settings were applied.

  • Producer - Broker section

    • Loss prevention: Message delivery is guaranteed through the acks=all option and retries settings.
    • Duplication prevention: The enable.idempotence=true option is used to ensure idempotency so that messages are stored only once even when retried due to network errors.

  • Consumer - Broker section

    • Loss prevention: Data loss during consumer restart is prevented through auto.offset.reset=earliest and Graceful Shutdown (immediate-stop=false) settings.
    • Duplication prevention: Duplicate consumption is prevented by explicitly sending an Ack after message processing is complete, through the AcksMode=MANUAL_IMMEDIATE setting.
high-availabilitykafkaaws-mskomsmessage-delivery-semanticsidempotence
Read originalShare on Threads

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.

Related

Olive YoungOlive Young Improves 'Preparing for Shipment' Cancellation Policy, Reducing Customer Service Inquiries by 90%Olive YoungOlive Young Reveals 'Intentional Over-Engineering' Case Study: Implementing CDC and Kafka in a Low-Traffic SMS SystemAWS Tech Blog (Korea)Celltrion Healthcare Achieves 70% Product Completeness in 4 Days with OMS Build Workshop Using AI-DLC and KiroYogiyoHow Trying to Resolve Deadlocks Led to Reducing LocksKT CloudCase Study: Building kt cloud's Deck, an Operational Data Platform Unifying Scattered DataWoowa BrothersWoowa Brothers Publishes Third Tech Blog Book: 'Modern Woowa Backend Development'