Kakao DKOS Reveals etcd Raft Algorithm and Maintenance Strategies for Kubernetes Live Upgrade

Kakao's Kubernetes as a Service, DKOS, adopts the Kubernetes Live Upgrade approach, which sequentially replaces master nodes to ensure zero-downtime application upgrades. Since this process involves repeatedly adding and removing etcd servers, understanding the Raft algorithm and maintenance principles at the core of etcd is essential for stable operations.

Raft Algorithm and Consensus Assurance

etcd uses a Replicated State Machine (RSM) structure to guarantee data consistency in distributed environments, implementing the Raft algorithm for this purpose. Raft ensures Availability, meaning servers respond even if some are down, and Safety, guaranteeing correct results.

• Leader Election: Servers exist in one of three states: Leader, Follower, or Candidate. If the Leader fails to send Heartbeats, a new Leader is elected after the Election timeout.
• Log Replication: Upon a Write request, the Leader replicates the log and performs a Commit once replication is complete for a Quorum (e.g., 2 out of 3 servers).
• Learner State: To reduce load on existing servers and maintain availability when adding a member, the new server initializes as a Learner state, excluded from quorum calculations, and is promoted to Follower after synchronizing logs.

Runtime Reconfiguration and Availability Management

Runtime Reconfiguration, which involves adding or removing servers during etcd cluster operation, is achieved through the existing log replication mechanism.

• Adding Members: New servers synchronize with existing servers by receiving Snapshots and logs. During this process, the quorum number changes, and using Learners prevents availability degradation due to synchronization delays.
• Removing Members: When the Leader receives a request to remove itself, it confirms log replication for a Quorum excluding itself and then Step Down to relinquish leadership. This ensures a new Leader is elected.
• Restriction: Raft allows only one member change at a time and rejects new change requests if uncommitted Config logs exist, maintaining consistency.

etcd Maintenance and Backup Strategies

Maintenance tasks such as log retention, compaction, and defragmentation are necessary to manage etcd performance and disk space.

• Log Retention: To prevent memory overflow, Snapshots are generated periodically and memory logs are Truncated. The default Snapshot generation interval is 100,000 log entries.
• Compaction: Compaction clears key change history (Revision) to reclaim disk space. Auto Compaction can be configured in Revision mode or Periodic mode.
• Defragmentation: Space freed by Compaction is not reclaimed on the actual disk, so Defragmentation must be performed. Note that Read/Write operations are blocked during this task.
• Backup and Recovery: Backup files containing integrity hashes are generated using the etcdctl snapshot save command. Kakao DKOS performs backups once a day to MetaKage storage and deploys a CronJob to retain data for the last 5 days.