Kakao Introduces 254PB Big Data Platform and Data Job Roles

Kakao has disclosed information about its big data platform, which processes 254PB of data, and its data job recruitment. Over 190,000 analysis jobs are performed daily across 140 shared Hadoop clusters and 9,100 data nodes.

Roles and Scale by Data Job Category

Data Platform Engineering designs and operates platforms that allow data scientists and engineers to handle data efficiently. Big data platform engineers evaluate and package tools in the Hadoop ecosystem, managing clusters via KHP(Kakao Hadoop Platform). This role functions as SRE (Site Reliability Engineering) in the data domain.

Database Platform Engineer manages user-facing data required for service operations. They are reducing dependency on commercial DBMSs like Oracle and transitioning to open-source bases, currently using MySQL and PostgreSQL as primary databases. Database engineers are responsible for SQL writing, modeling, backup and restoration, and monitoring automation.

Recruitment and Competency Requirements

The recruitment process prioritizes actual competency and growth potential over master's or doctoral degrees. The proportion of personnel with master's or doctoral degrees among those who joined in the last two years was low, and non-majors or those with bachelor's degrees are fully eligible to apply. Coding tests have no language restrictions depending on the applied field, allowing candidates to take the test in the language they are most confident in.

Required competencies by job role are as follows:

• Data Engineer: Data pipeline, distributed systems knowledge, programming skills
• Data Scientist: Mathematical/statistical expertise, ML/AI modeling, business domain understanding
• Database Platform Engineer: SQL, DBMS modeling, ability to handle cloud environments
• Big Data Platform Engineer: Java proficiency, Hadoop ecosystem knowledge, willingness to operate platforms