Experiment design know-how every junior designer should know

When I took on my first task of increasing the sign-up conversion rate for non-members, the first problem I ran into was deciding where to start improving in a situation with multiple drop-off points. The consent screen and ID verification screen were strongly shared-module in nature and required legal/compliance review, making experiments slow, whereas the intro screen could be iterated on quickly and also had a large impact on overall conversion. So, weighing speed and impact together, I decided to start improving from the intro screen.

First I reviewed existing experiments, and learned that many attempts had already been made, and quite a few cases had not clearly beaten the existing version. Instead of just looking at win/loss, I had to read what problem each experiment started from, why that hypothesis was set up, and how the experiment design was structured to validate it. The less experiment experience you have, the more important it is to spend time structurally interpreting existing learnings rather than rushing to come up with new ideas.

The first experiment was a version emphasizing an agent and choices, but it failed. The hypothesis that "fewer choices will raise the conversion rate" was vague, and in reality the existing version already had a single CTA button, so the new version effectively just added more buttons. I hadn't sufficiently looked at why the user entered this screen or whether a recommendation was really needed before setting up the hypothesis, so it fell apart—after that, I switched to analyzing the existing context first, before building a screen.

Next, rather than rushing to create a new draft, I clearly identified the problems with the existing intro screen. The copy wasn't appealing, so it didn't reveal what the user was actually interested in, and image loading was slow, taking 2-3 seconds on low-spec devices. Reflecting past learnings that keywords like high interest rate products and daily interest payout were effective, I changed the copy, and also improved loading speed by applying up-to-date graphics and a low-capacity file extension. As a result, just changing the copy and images led to both click-through rate and conversion rate rising together.

After that, I experimented with expressions that let users picture a concrete scene. Instead of copy centered on feature descriptions, I changed it to first show the situation where users would feel the benefit fastest, and as a result CTR rose by 5% and CVR also improved significantly. This confirmed that even with the same content, the first impression can change greatly depending on how it's expressed, and led to the conclusion that every element shown on screen needs to be examined more carefully.

The key points are three:

• Pick the section where you can experiment quickly first.
• Read why a result happened, rather than just the win/loss of past experiments.
• Set hypotheses sharp enough that you validate only one thing at a time.

There were many failed experiments, but the clearer the hypothesis, the less the direction wavered, and even failures became hints for the next experiment. An experiment isn't a process for confirming success or failure—it's a process for making the next choice faster and clearer. When you're just starting out, rather than a grand answer, starting small but setting a sharp hypothesis matters more.