How Does a 1:1 Video Chat Service Automate E2E Regression Testing

Azar automated repetitive regression tests using Pytest and Appium. The core challenge was that, unlike ordinary apps, interaction with another user—as in 1:1 video chat—is part of the test itself.

To solve this, tests were divided into Non-interaction and Interaction categories. For Interaction tests, the design created 2 drivers within a single test and enabled execution across platform combinations expanded to Android-Android, Android-iOS, iOS-Android, and iOS-iOS.

Two problems were blocking parallel execution.

• Due to random matching, accounts from different test scenarios could get mixed and matched with each other
• Test steps that didn't need to depend on the counterpart's action were still executed sequentially at every step, increasing execution time

The former was resolved with cooperation from the server development team, which provided a segment-based matching feature. After creating test accounts, segments were assigned to main accounts and sub accounts so that only matching tests would be paired with each other.

The latter was addressed by using ThreadPoolExecutor inside the tests so that mutually independent actions could run simultaneously. Parallelizing dependency-free tasks such as login, connection, and some verification steps reduced execution time per case by 1-2 minutes.

Platform-specific differences were absorbed using an interface-based Page Object Model. In the test code, page objects are handled through a common interface type, while the actual implementation branches to platform-specific behavior simply through driver injection, improving maintainability.

On the execution side, pytest_addoption and pytest_generate_tests were used to create parameter groups for each platform combination, allowing only the necessary combinations to be selectively run or skipped. As a result, a system was built to run 308 test cases across 4 combinations, and work that would take about 102 hours manually was reduced to about 7 hours through automation.

This allows repetitive nightly regression tests to be handed off to automation, letting the QA team focus more on exploratory testing of new features during working hours. Challenges that remain include backward version compatibility, further reducing execution time, improving image/audio verification, and advancing AI-based verification logic.