Connecting HDFS Clusters at 1EB+ Scale with Different Histories: Challenges and Design Decisions

LY Corporation operates a large-scale data platform with a total capacity of 1EB (exabyte) or more to support various services such as search, e-commerce, and messenger. This platform was formed by combining the major clusters of the former LINE Corporation and Yahoo Japan Corporation, and in the process of integrating the two organizations' different design philosophies and operational models, it faced two major challenges.

The first is the operational challenges arising from the surge in data. As data scale grew, scaling constraints and operational bottlenecks such as Capacity management, NameNode load, and network impact had to be addressed.

The second is interconnection between data platforms resulting from organizational integration. The key was designing connection endpoints, access control units, and data transfer paths to connect platforms with different design approaches so that data could be mutually utilized.

In the past, LINE built web portal-based permission management and a data catalog to enhance user convenience for data utilization across various departments, while Yahoo Japan evolved its platform from initial analysis for specific purposes into a company-wide foundation that is easier to operate stably.