What is A/B testing?
A/B testing is a controlled experiment on a website or in an app. Traffic is split at random between a control (A) and a variant (B). You measure a goal metric defined in advance, for example the conversion rate.
Randomization is the key. Because both groups are made up the same way, you can trace a difference in results back to the change. Seasonality, weather, or a running campaign affect both groups equally. That is exactly what separates an A/B test from a before-and-after comparison.
The principle is old. Medical studies have worked this way for decades. Online, it became standard in the 2000s, when Amazon, Google, and Booking.com began testing decisions systematically. Booking.com is considered one of the companies with the most experiments running in parallel today.
A/B test, split URL test, multivariate test: what's the difference?
Test type | What is tested | Traffic needed | Typical use |
|---|---|---|---|
A/B test | One control against one or more variants on the same URL | medium | new argumentation, new layout of a section |
Split URL test | Two completely different pages on separate URLs | medium | new landing page, new checkout |
Multivariate test | Several elements at once, in all combinations | very high | fine-tuning with a lot of traffic |
A/A test | The same version twice | medium | checking whether the tool and tracking measure cleanly |
For most websites, the classic A/B test is the right choice.
A multivariate test with three headlines and three images has nine combinations. Each one needs enough visitors for a reliable result. For most websites, that is too much.
There are also bandit methods. They shift traffic to the better variant during the test. That brings more revenue in the short term but delivers less clean insights. For promotions with a short run time, that can make sense. For decisions that should last longer, the classic A/B test is better.
When A/B testing pays off
Many people check too late whether their traffic is enough at all. How many visitors a test needs depends on three values. First, today's conversion rate. Second, the effect you want to detect. Third, the certainty you want. The usual choice is a 95% confidence level and 80% statistical power.
Starting point | Effect to detect | Visitors per variant | Total visitors (A and B) | Test duration at 5,000 visitors per day |
|---|---|---|---|---|
3% conversion rate | +10% (to 3.3%) | about 53,000 | about 106,000 | about 3 weeks |
3% conversion rate | +20% (to 3.6%) | about 14,000 | about 28,000 | about 6 days, rounded up to 2 weeks |
Worked example with a 95% confidence level and 80% statistical power, two-sided test.
Two things stand out. Small effects need a lot of traffic. For half the effect, you need about four times as much. And every test should run for at least two full weeks, even if the number is reached sooner. Otherwise weekdays and paydays skew the result.
To work out your own numbers, divide the required visitors by your daily traffic and round up to full weeks.
As a rough rule of thumb, classic A/B testing gets difficult below about 1,000 conversions per month on the path you are testing. Then you are better off with bigger changes that have a bigger effect, tests on intermediate steps like “Add to cart,” and qualitative methods such as user interviews.
The A/B testing process in six steps
1. Analysis. Find out where and why visitors drop off. Web analytics numbers show you where. User interviews, surveys, and session recordings show you why.
2. Hypothesis. State what you change, why it should work, and how you measure success. A good hypothesis builds on observed behavior and not on taste. A simple template: Because we observed X, we expect change Y to cause effect Z, measured by metric M.
3. Prioritization. Rank your ideas by expected value contribution, effort, and traffic needs. That way the idea with the biggest payoff runs first.
4. Implementation and quality assurance. Build the variant in the testing tool or on the server side. Check it in all important browsers and on all important devices. A bug in a variant costs real revenue. Also check that tracking records the goal metric correctly in every variant.
5. Test duration. Let the test run for the duration you calculated beforehand, and don't look for a winner every day. Anyone who stops as soon as one variant is ahead will find plenty of false winners.
6. Evaluation and rollout. Check statistical significance, sample distribution, and segments. Roll out the winners and document what you learned. Even a lost test tells you something about your customers.
What you can test
Area | Typical hypothesis | Goal metric |
|---|---|---|
Product page | Visitors who see delivery time and returns right next to the price hesitate less. | Add-to-cart rate |
Checkout | A guest checkout without mandatory registration reduces abandonment. | completed orders |
Form | Fewer required fields bring more inquiries without lowering quality. | inquiries and qualified leads |
Pricing page | A highlighted middle option steers the choice. | revenue per visitor |
Landing page | A headline that names the visitor's problem keeps them on the page. | clicks to the next step |
Examples of hypotheses by page type.
The reasoning matters more than the element. A “red versus green button” test without a hypothesis rarely delivers anything. Changes that address a customer concern or use a behavior pattern such as loss aversion or social proof work better. Our Behavior Pattern Model turns such patterns into testable hypotheses.
Statistics without the jargon
Confidence level. It tells you how sure you can be that the difference is not random. 95% is the usual choice. That also means: out of 20 tests with no real effect, one will show a winner anyway on average.
Statistical power. It tells you how likely your test is to find a real effect. 80% is the usual choice. A test with too few visitors has low power and misses winners. In an analysis of 115 A/B tests by the statistician Georgi Georgiev, nearly 70% were underpowered.
Sample size and test duration. You set both before the start, not during the test. Plan them for your actual setup, including multiple variants and unequal traffic splits.
Frequentist or Bayesian. The frequentist analysis answers whether a difference is statistically significant. The Bayesian analysis answers how likely which uplift is and what it means in euros. For management decisions, the second is often easier to understand. It turns a result into a range of likely uplifts that you can translate into revenue.
To check whether a finished test is statistically significant, run a significance test on both variants at the confidence level you set beforehand.
How often tests win
Many tests produce no significant result. That is normal and not a sign of poor work. The win rate depends mostly on where the hypotheses come from.
Anyone who tests best practices from blogs rarely hits the mark. Anyone who first understands why customers hesitate hits more often. At freenet, two thirds of all tests were successful, and the cumulative uplift was 30%. The reason was not a better tool but hypotheses based on customer understanding.
Even a test without a winner has value. It keeps an expensive change with no effect from going live. And it shows you which assumption about your customers is wrong.
Examples from practice
Shoe online shop: the winner cost more than expected
A fashion retailer tested three variants of the product page for one shoe model against the control. One variant emphasized discounts, one emphasized quality, and one improved the presentation. The winning variant raised the add-to-cart rate by 244% and orders by 43%. At the same time, almost three times as many returns came back. After returns, 22% more contribution margin remained.
So the test was a success, but only half as big as the first number suggests. Anyone who looks only at the conversion rate flatters such tests. The whole story is in the case study on conversion, returns, and contribution margin.
Fashion D2C brand: testing as a growth lever
The market for the core business of a midsize D2C brand was saturated. More ad budget hardly brought in any new customers. Together with the internal team, experimentation became a fixed part of every important decision. The result: +20.69% revenue growth and €22.8 million value contribution per year. The values are calculated conservatively, with a confidence interval and depreciation. You will find more results on our page on conversion optimization.
Common A/B testing mistakes
Stopping too early. Anyone who stops the test as soon as one variant is ahead significantly raises the risk of a false winner.
Sample too small. Without calculating it beforehand, it stays unclear whether the test can find an effect at all.
Uneven split. If one variant gets significantly more or fewer visitors than planned, something is wrong with the allocation. Experts call this sample ratio mismatch. The result is then unusable.
Flicker. If the variant visibly loads after the original page, it distorts behavior.
Too many goal metrics. Anyone who measures ten metrics almost always finds one that wins by chance. Set one primary goal metric beforehand.
Novelty effect. Returning visitors click on anything new at first. The effect can disappear after two or three weeks.
Wrong goal metric. More orders do not automatically mean more profit, as the shoe test shows.
A/B testing, SEO, and data privacy
Google explicitly allows A/B tests. You should follow three rules. Don't show search engines a different version than visitors, because that would be cloaking. Set a canonical tag pointing to the original page for split URL tests. And end tests after the planned duration instead of letting them run for months.
On data privacy: most testing tools set cookies or store an identifier in the browser. For that, you usually need consent through your consent banner. Server-side tests without personal data are easier to secure. Clarify the integration with your data protection officer.
Tools for A/B testing
Since Google Optimize was discontinued in September 2023, most companies use specialized providers. Roughly speaking, there are two groups. Client-side tools change the page in the browser. They are quick to set up and well suited for marketing teams. Server-side tools deliver the variant directly from the server. They take more effort, but they are faster, more stable, and better for tests on prices, search, or logic.
Which tool fits depends on traffic, tech stack, and data privacy requirements.
Do it yourself or with an agency?
For your first tests, a tool, a clean hypothesis, and a calculated test duration are enough. It gets harder when A/B testing should bring in revenue permanently. Then you need research, statistics, development, and an organization that also acts on the results. What to look for when choosing a CRO agency is covered in a separate guide. If you lack the capacity or experience, you can get A/B testing as a service from us. How A/B testing fits into overall conversion optimization is explained in our conversion optimization guide.