Finding a CRO agency: criteria, costs, and the right questions

For over twenty years, I have been looking at A/B tests. Many of them were our own. Hundreds came from other agencies and service providers. The stories clients tell me afterward sound strikingly alike. An agency reports winners month after month, and none of it shows up in revenue. Or the testing tool happily counts conversions that never happened in the shop.

At some point, leadership stops believing in the whole method. Whoever comes next works on scorched earth.

I'm not pointing fingers at others here. We have made every mistake in this guide ourselves and had to learn what causes it. Today we also work for large corporations, and there we can no longer afford such startup mistakes.

This guide collects how to recognize a good agency. Yes, we offer conversion optimization ourselves. You can still put these criteria to any provider, including us.

Do you need a CRO agency at all?

The honest answer: not necessarily. It depends on your traffic, your budget, and what your team can already do.

One question gets skipped in pitches. Is your traffic enough for tests at all? For a meaningful A/B test, you need several tens of thousands of visitors per variant, depending on the conversion rate. You can work out in two minutes whether that's enough for you: estimate the test duration from your daily traffic and your baseline conversion rate. If it isn't, research without tests is often the better investment. No agency likes to hear that, and neither do we :-)

Agency or consulting: what the difference means

At some point in every first call, someone says: “We're not a classic agency, we also consult.” Almost everyone says it. Behind the two terms are really three services, and good providers cover all three.

Anyone who buys only one of the three services often notices the gap only after months. Without strategy, the team tests in the wrong place. Without implementation, good recommendations end up in a drawer, right next to the last strategy paper. Missing enablement is the most expensive gap, because then progress ends on the day the contract expires.

So ask less about the label. Ask instead which of the three services a provider really delivers itself.

8 criteria for choosing

How do you recognize a good agency before you sign? Reference logos don't help much, because everyone has them. The following eight questions help more.

1. Who does the implementation?

In the pitch, a complete team sits on the slide. In practice, many agencies do the concept themselves and hand development to subcontractors. Sometimes these are freelancers who have never met the client. That looks efficient. But every handoff costs time, and with every handoff, a bit of understanding of the actual problem gets lost. It's a game of telephone, billed by the hour.

Question to ask: Who writes the test code? Are the developers on your own team?

2. What do the hypotheses rest on?

In many projects, hypotheses come out of a workshop, with lots of sticky notes and even more conviction. Good hypotheses come from data about your customers and from experience with earlier tests. An agency with a large, documented test database knows which patterns hold in your industry and which only looked good in the workshop.

Question to ask: How many tests have you documented? Can you show anonymized examples from my industry?

3. How is AI used?

Hardly any pitch gets by without an AI slide today. Rightly so, because AI speeds up almost every step: data analysis, evaluating customer feedback, hypotheses, test development, and reporting. But it replaces neither tests nor judgment. An AI formulates twenty plausible hypotheses in seconds. Which of them is true for your customers, it still doesn't know. Only the test shows that.

Some tool vendors now go a step further. You describe an idea in one sentence, the AI builds the variant and starts the test. The message behind it is clear: soon you won't need agencies anymore.

For test development, that is partly true. An AI builds simple variants faster than any developer today, and we use that ourselves. The only catch is that development was rarely the bottleneck. The bottleneck is the question of what to test in the first place. An AI inside a testing tool knows your page and your clicks. It doesn't know your customers. It doesn't know why someone hesitates in checkout, what your competitor does better, or which goal your leadership is pursuing right now.

When a test costs only one prompt, a volume problem also arises. Instead of five tests, soon fifty are running. Each of them still needs enough visitors, though. More tests on the same traffic mean shorter or smaller tests and therefore more chance winners, see criterion 5.

That leaves the question of who checks the AI. If the AI in the tool writes the hypothesis, builds the test, and evaluates the result, the system checks itself. The vendor has little reason to change that. After all, it sells the tool.

Questions to ask: What do you use AI for? Who checks the result before it reaches me? Where do the ideas come from that the AI implements?

4. How fast does the first test go live?

Every week without a test is a week without insight, and it costs revenue. Still, many projects start with an audit that takes several months. That looks thorough. In the end, there's a thick PDF, and the best hypotheses emerge only once the first tests are running.

Questions to ask: How many days pass from kickoff to the first test? How many tests per month are realistic?

5. How are tests planned and evaluated?

Now it gets briefly technical, but it's worth it. In statistics, good providers separate from weak ones more clearly than anywhere else. Errors stay invisible here for a long time. They only show up when the promised uplift never arrives in the shop. We see three mistakes especially often.

Stopping tests too early. Anyone who stops tests as soon as the tool reports “significant” produces false winners. According to an analysis of 115 A/B tests, nearly 70% were designed too short to reliably detect a real effect.

Hunting for winners in segments. A test brings no uplift overall. Then the result is sliced by segment until one wins: mobile, new customers, visitors from the newsletter. You might think now: “But those are exactly the segments we need for personalization.” The idea is right, the method isn't. Every additional comparison raises the probability of finding a winner purely by chance. Statisticians call this alpha error inflation. With ten segments and a 95% confidence level, the chance of at least one false winner is already around 40%. On top of that, the sample is almost never big enough to compare individual segments. That's how uplifts come about that don't exist. If you want to compare segments, you have to plan it before the test and size the sample for it.

Choosing the wrong metric. In case studies from other providers, you often find uplifts on revenue per visitor even though the client's traffic wasn't enough for that. The metric sounds like the most important one. The catch is that revenue values fluctuate much more than conversions. A test on revenue therefore needs many times as many visitors. If traffic isn't enough, the result is chance. Methods like CUPED reduce the variance, but they don't replace a sufficient sample. You see how big the difference is when you size the same test once for conversions and once for revenue per visitor.

Questions to ask: Do you calculate sample size and test duration before the start? Do you plan segment comparisons in advance or look for them afterward? Which metric do you test on, and is my traffic enough for it?

You can check the answers yourself: estimate the test duration from your traffic and baseline conversion rate before the test, and run a significance test on the result after it.

6. Is the data even right?

Every evaluation is only as good as the data underneath. And that data is wrong more often than you'd think. The testing tool counts conversions differently from the shop. Consent banners swallow part of the visitors, and not equally in both variants. One variant loads visibly later and flickers. Or visitors aren't distributed across the variants the way they should be.

Errors like these appear on no dashboard. You only find them when someone checks the numbers from tool, analytics, and shop backend against each other. For that, an agency needs technical competence and a certain independence. Anyone who earns a share of a tool's license rarely looks critically over the tool's shoulder.

This applies to the tool vendors themselves, too. Many now offer test implementation as well, often at a price no agency can match. For many companies, that's understandably attractive. You should just know what you get for it. The vendor earns from the license, so the tool should look as irreplaceable as possible. What gets built is what works well in the tool. Whether your problem can be solved with a test at all is rarely asked. And whether its own tool measures wrongly, the vendor checks about as thoroughly as a baker judging his own bread.

Question to ask: How do you check whether the data from the testing tool and analytics is right? Do you earn a share of tool licenses, and which tools do you work with?

7. Does the work start with the customer?

Many testing programs start with tactics like button colors, trust seals, or countdown timers. They're sold as best practices, but it's really trial and error. Some tactics work. Which ones, nobody knows beforehand, not even the agency with the longest list.

The bigger problem lies elsewhere. Every company faces its own challenge in the market: different customers, different competitors, different reasons why someone doesn't buy. A list of tactics misses that core. It answers questions nobody asked and skips the one that matters. Why don't your customers buy?

Question to ask: How do you find out what keeps my customers from buying?

8. Which research methods are used?

No single method delivers a complete picture. Analytics shows where users drop off, but not why. Interviews show why, but not how many customers it affects. Anyone who does only one of the two sees half and takes it for the whole. Good agencies therefore combine methods, for example interviews, usability tests, onsite surveys, and analytics.

Questions to ask: Which methods do you use? May I see an anonymized research report?

What does conversion optimization cost?

That is the question almost every agency page dodges. I'll try anyway. There are no fixed market prices. Costs depend mainly on three things: how many tests run per month, how much research is included, and whether the agency does its own development.

Before any contract, there should be a thorough analysis that clarifies your problem. Serious providers do it before they write a quote. It also shows whether optimization is worth it for you at all. Sometimes the honest answer is no.

Always weigh the costs against the possible effect. An example: A shop with €10 million in annual revenue that makes 2% more revenue through optimization gains €200,000 a year. A proof of concept over 6 months at €5,000 a month costs €30,000. At €1 million in revenue, the same calculation looks very different. Then someone should tell you that, too.

11 questions for the first call

1. Who on your team will work on my project specifically?

2. Who develops the tests?

3. How many tests have you run in my industry?

4. How does a hypothesis come about at your company?

5. Which research methods do you use?

6. How do you calculate the duration of a test, and which metric do you test on?

7. How long does it take to get to the first test?

8. How do you use AI, and who checks the result?

9. How do you check whether the data from the tools is right?

10. What does a test report look like at your company?

11. What does my team learn from working with you?

Warning signs

• Guaranteed uplifts. Nobody can know beforehand how a test will turn out. Anyone who promises it anyway is selling hope.

• Tests without duration planning. Stopping by gut feeling leads to false winners.

• Winners from segments after the fact. Anyone who filters after the test until a segment wins is selling chance as a result.

• Revenue as the test metric with too little traffic. Revenue per visitor needs many times as many visitors. Without the necessary sample, every result is unreliable.

• Blind trust in the tool. Anyone who never reconciles numbers from the testing tool with analytics and the shop may well be evaluating measurement errors.

• Implementation as a freebie with the license. Cheap test development from the tool vendor replaces neither research nor strategy nor an independent check of the data.

• Hypotheses from tactic lists. What worked for others doesn't have to work for you.

• “AI replaces tests” or “AI replaces research.” AI builds tests faster. It doesn't reliably decide what should be tested or whether the result is right.

• Success fee on every uplift. It rewards short tests and pretty numbers instead of reliable results.

• No insight into earlier work. Anyone who can't show an anonymized report usually doesn't have one.

The first 90 days

The order matters. Research and tests run in parallel. Anyone who analyzes for three months first and then tests loses time without getting better hypotheses.

Checklist

• Traffic is enough for tests (checked by estimating the test duration)

• Implementation is done by the agency's own team

• Hypotheses come from research and documented experience

• Test duration is calculated before every test

• Data from tool, analytics, and shop is reconciled

• Segment comparisons are planned in advance, not searched for afterward

• The test metric fits the traffic

• The first test is live after about two weeks, and research runs in parallel

• AI is used and checked

• You have an anonymized report in hand

• Costs are in a realistic proportion to the possible effect