Building a Data Analytics Library (Part 2) - Integrated Testing and Documentation Strategy

DevPlay Analytics is a Python library that uses PySpark to analyze data in Amazon S3. Unlike typical libraries, data analytics libraries depend heavily on S3 I/O and the PySpark environment, which makes building a testing environment challenging.

As a testing framework, both unittest and pytest can be considered. unittest is class-based and manages resources through setUp and tearDown. pytest, on the other hand, offers concise function-level syntax and powerful fixture functionality, making it more efficient for developing data analytics libraries that need to control the creation and teardown timing of complex resources.

However, since the IO layer depends on PySpark and S3, there are limits to how much of it can be mocked. Initially, the approach was to directly prepare a test bucket and sample data in S3, but due to the nature of the business logic, configuring tests this way remains difficult.