Data Analytics Library Development Story (1)

Devsisters' DevPlay Data Platform team is responsible for engineering to ensure product logs and data flow smoothly. This post introduces the development process of the DevPlay Analytics library, which helps data analysts use PySpark SQL more conveniently.

Currently, DevPlay processes big data through parallel distributed processing via Apache Spark, with analysts analyzing data stored in AWS S3 on Jupyterhub and managing workflows with Apache Airflow.

However, as the platform grew in scale, the following problems arose:

• Complex data paths: The inconvenience of having to manually enter numerous S3 paths depending on the game and data layer.
• Configuration value management: Difficulty in managing credentials and configuration values needed for data access.
• Inconsistency across environments: Development cycle delays caused by the inability to share utility functions between the Jupyter Notebook and Airflow batch environments.
• Low analyst friendliness: Usability issues inherent to PySpark itself.