Kakao Releases 'khp-package', a Release Management Tool for Its Proprietary Hadoop Platform KHP

Kakao has developed and applied khp-package to manage open-source Hadoop component compatibility and releases for its proprietary Hadoop platform, KHP (Kakao Hadoop Platform). This tool addresses version dependency and compatibility issues among open-source Hadoop components and automates the release process of distribution packages.

Why We Developed It In-House Instead of Using Apache Bigtop

Kakao evaluated Apache Bigtop for Hadoop component packaging but chose in-house development due to issues where test code became dependent on component versions because of compile-time dependencies. Bigtop incurred high maintenance costs because test code was significantly affected by component version changes due to compile-time dependencies. In contrast, khp-package is optimized for Kakao's infrastructure environment, reducing the burden of modifying test code when component versions change and facilitating mutual compatibility verification.

Key Features and Workflow of khp-package

khp-package includes commit-level pointer management for Hadoop components, testing procedures, and scripts for automated deployment and releases. The main steps are as follows:

• Integration Testing: Compatibility is verified after deployment via khp-ansible on a 'live cluster' identical to the production environment. CLI and functional tests are performed on major components such as Hadoop, Spark, Trino, and Hue, with the entire test taking over 140 minutes.
• Build and Packaging: Build tools for each component (Maven, Python, etc.) are standardized and executed via rake tasks. For Anaconda, Docker is used to configure conda environments for each OS, and packages are created as tar.gz files.
• Deployment: Binary artifacts are uploaded to an internal file server. Strict version control is enforced by preventing overwrites for non-SNAPSHOT versions.

Release Management and Artifacts

Verified releases are organized in the khp-release repository with release notes (.md) and download links (.yml), then published to the internal Wiki. KHP versions are tagged in the format khp-<x.y.z>-hadoop<major version>, while individual components such as Hadoop, Spark, and Anaconda are managed with filenames that include their own build dates or patch numbers. This structure provides compatibility verification and operational efficiency optimized for Kakao's infrastructure, compared to general-purpose open-source tools.