Kubernetes Pod Clustering Implemented with Lease and Hazelcast

Kubernetes is fundamentally a stateless architecture with no data sharing between Pods, but by leveraging service discovery and clustering, only the functions that need it can be cooperatively grouped together.

Two improvement cases were applied to CrashReport.

• Real-time 5-minute statistics: Previously, GCP Dataflow and a separate Kubernetes cluster had to be used, which came with the burden of learning Apache Beam, along with version management and end-of-support issues.
• Symbolicating cache sharing: Since caches accumulated separately per Pod couldn't be shared, there were limits to performance improvement.

The real-time statistics feature was rebuilt using Apache Camel and Spring Boot. Inside the Pod, it was split into a streaming router for pre-aggregated statistics calculation and a router for final statistics calculation, creating a structure similar to MapReduce. Camel's Aggregation Strategy was used to sum up messages, and leader election using Kubernetes Lease created an Active-Standby configuration.

For Lease-based clustering, application configuration and ServiceAccount permission settings were required. After configuration, normal operation was verified through the cluster member list and leader change logs, and the leader could also be checked with kubectl get leases leaders-lock2 -o yaml.

As a result of the implementation, while maintaining stability similar to the existing configuration, the number of unacknowledged messages and the age of the oldest unacknowledged message decreased.

Hazelcast was introduced for symbolicating cache sharing. The existing embedded H2DB-based JPA cache was difficult to share between Pods, but using Hazelcast's Key-Value cache and Kubernetes auto discovery made it possible to share data across the cluster. Account and permission settings along with communication port settings were added on top of this to configure the cluster.

After deployment, operation was verified through Hazelcast clustering success logs, and as the cache was shared to nearby Pods, the average symbolicating time decreased. Additionally, the age of unacknowledged messages also improved by an average of about 50% based on the maximum value. Thanks to Hazelcast's cache synchronization option, existing caches were preserved even during new deployments, reducing initial performance degradation as well.

Ultimately, the key point is that while maintaining Kubernetes' fundamental stateless nature, layering cooperative structures between Pods using open source tools like Lease and Hazelcast can boost both real-time processing and cache performance together.