It seems these days everybody and their grandmother is selling a conversion rate optimization (CRO) tool to online businesses. But recently, I have run across too many claims that mistake correlation for causation when it comes to advertising CRO benefits. This is a dangerous mistake to make, because it means your business may adopt a tool that doesn’t deliver as promised. Too often I see references to correlation-based claims as if they are convincing, when in fact, they are fool’s gold. Here are some correlation-based claims that can lure you into thinking a CRO tool causes purchase increases:
“People who use our tool are 80% more likely to make purchases…”
“Using our tool results in an 80% increase in average order value…”
“People who use our tool buy at an 80% increased rate…”
“After adopting our tool, our customers see an 80% increase in sales…”
Do claims like these look familiar? Do they sound convincing? Are you falling for them? The problem with such claims is that they only reflect correlations, not causation. You may have heard this distinction before, but if you remain somewhat fuzzy, I cannot stress enough how important it is to understand the difference. Your business could depend on it.
Correlation vs. causation simply put
Simply put, correlation indicates events that tend to happen together. Take the first example above. The correlation is between two aspects of people’s behavior: using a tool, and making a purchase. It might seem like one leads to another, but there is absolutely no reason to believe that. With correlation, there is often another unknown cause that’s actually driving both events. For example, perhaps the online shoppers who made purchases were more motivated to purchase in the first place, and this motivation compelled them to also use the tool. Without an experiment, we have no way of knowing the likelihood of such shoppers making a purchase without the tool. Maybe the rate would have gone up!?
SEE ALSO: BREAKTHROUGH WAYS TO CUT CUSTOMER SUPPORT COSTS
Causation, on the other hand, is genuine cause-and-effect, meaning one thing has a direct impact on another. Causal evidence gives a clear indication that the chances of seeing the same effect in multiple places are good. To demonstrate causation, a true experiment is needed. Before believing the claims of a CRO vendor, you should know whether those claims come from a true experiment or not.
So what is a true experiment?
When it comes to CRO, a true experiment is most commonly an A/B test, which psychologists describe as a single-factor two-level test. To be true experiments, it is important that A/B tests have these properties
One condition includes the CRO tool; the other condition does not
Besides the presence or absence of the tool, nothing else on the site changes between conditions
The two conditions are run concurrently, not serially, to avoid confounds
New website visitors are randomly assigned to either condition and remain in that condition for the duration of the test
Conversion rates are measured for both conditions regardless of whether the CRO tool was actually used
From a test like this, the important claim can be made:
“Our tool causes a 20% increase in conversion rate…”
That’s the best and only claim any vendor should need to make, because if it’s true, an online retailer can have confidence that adding this tool will move the needle. (Maybe not 20%... maybe even more…)
How should increases in conversion rate be measured?
Rates are percentages, and percentage increases of percentages can sometimes confuse people. In a true experiment, if 20 of every 100 visitors makes a purchase without the CRO tool present, and 35 of every 100 visitors makes a purchase with the CRO tool present
(regardless of whether it is used or not), then we are dealing with a 20% conversion rate increasing to a 35% conversion rate. What is the percent increase? The math is easy enough:
percent = (new rate – old rate)/ (old rate).
Or in our example, (.35 - .20) / (.20) = .75, or a 75% increase in conversion rate. Intuitively, that’s because 15 percentage points represent 75% of 20 percentage points. That would be a wonderful tool!
If a vendor can’t produce solid evidence, don’t buy
If a CRO vendor can’t produce solid evidence of the CRO uplift caused by their tool, don’t buy their tool. If all you hear are correlational results, then be concerned. It probably means the vendor doesn’t know how to measure the effects of their tool, or has tried to measure these effects but didn’t get good results. It may also mean that the vendor doesn’t realize the difference between correlation and causation, and sees no need for a true experiment. Whatever the case, solid CRO tools should produce solid evidence of uplift.
SEE ALSO: YOU’RE DOING IT WRONG: ART VS SCIENCE IN CUSTOMER SUCCESS
At AnswerDash, our tool provides a contextual Q&A layer atop websites and web applications to give end-users answers where and when they need them. We provide an A/B testing tool that enables our customers to run true experiments to measure how AnswerDash lifts conversion rates for e-commerce sites. Our A/B testing tool also enables our customers to measure how AnswerDash reduces customer churn for SaaS applications. We do produce correlational results along the way, but our focus is always on causal results from true experiments. When we tell a customer we can increase their business by 10%, we are confident in the causal data we use to make this claim.