The Failures and Successes of Automating Test Cases with Gemini

Drafting test cases (TC) for every new task is essential, but reading through a spec and breaking it down into detailed cases and documenting them takes a lot of time. Musinsa's QA team experimented with using Gemini to streamline this repetitive work, testing whether people could instead focus on more important review and automation tasks.

They started with the relatively simple membership tier calculation policy, and even with just a prompt-based approach, extracted 20 to 38 meaningful TCs. Building on this success, they expanded to more complex tasks like coupons, payments, and global product detail pages, but the results were mostly failures.

The causes of failure were clear.

• When a long spec document was fed in all at once, key conditions in the middle were dropped due to the Lost in the Middle phenomenon.
• Even when pre-set rules were included in a Gem, the AI often prioritized the immediate instruction over them, frequently ignoring baseline policies or the line break () rule needed for Excel.
• When too many conditions were crammed into a single prompt, the AI would misunderstand core policies or break the output format.

As a notable example, the payment and coupon task yielded only 102 out of the needed 299 cases (34%), and the inventory task produced solid UI TCs but missed the critical stock exception cases by option, generating only 46 out of 96 cases. In the global task as well, having too many conditions caused the AI to misinterpret core policies.

The solution wasn't a stronger prompt, but a split extraction process.

1. First, have the AI learn the QA guide and the spec document so it understands the basic rules and policies upfront.
2. Rather than generating TCs immediately, first extract a component table of contents for what needs to be tested, and have a human check for omissions.
3. Then generate TCs one component at a time, enforcing TSV format for Excel compatibility.

With this approach, the amount of information the AI had to process at once was reduced, greatly improving the depth and accuracy of the TCs. In one case, the number of extracted cases went from 33 without component separation to 104 after separation.

The results were also confirmed numerically. Out of a total of 70 tasks, AI was used to design TCs in 57 (81.43%), and 82.6% of all TCs written were generated via AI. Even with the added time for learning guides and specs and for review, the overall schedule was shortened by 39.49%.

Ultimately, the key was not handing everything to the AI at once, but rather teaching it the rules, having it build a table of contents, and giving sequential instructions in small units. Through this process, Musinsa's QA team established AI not merely as a simple drafting tool but as a practical partner that reduces repetitive work, and plans to further expand into a more stable automation pipeline going forward by also leveraging the Claude API.