RabbitMQ Classic Queue Memory Failure and the Transition to Quorum Queue

During a large-scale coupon issuance process involving 15 million cases, a failure occurred in which message processing stopped due to excessive memory usage in the RabbitMQ cluster. As a specific node switched to an unsynchronized state, message consumption completely halted to preserve data integrity.

The structural limitations of Classic Mirrored Queue were the main cause. When memory shortages delayed responses during the synchronization process between the Master node and Mirror nodes, RabbitMQ halted message processing until synchronization was resolved, in order to ensure data safety. This acts as a critical weakness in terms of availability.

The following process was followed to respond to the failure.

• Emergency Recovery: Instead of the existing broker, a new, independently operated RabbitMQ broker was created, normalizing the service within 30 minutes.
• Cause Reproduction: By combining a large influx of messages with a forced broker restart scenario in the development environment, the team succeeded in reproducing the unsynchronized state.
• Direction for Resolution: The RabbitMQ team itself also acknowledges the limitations of Classic Mirrored Queue, and the team aims to secure stability by transitioning to the recommended Quorum Queue.