Incrementality Testing: Proving What Really Drives Sales

Here is an uncomfortable thought experiment. You run a retargeting campaign that shows adverts to people who already visited your site. The report glows: hundreds of sales credited to those adverts. You feel great. But pause for a second. Many of those people had already added items to a basket. Would they have bought anyway, advert or no advert? If the answer is yes, then your campaign did not create those sales; it just took the credit for them. Incrementality testing exists to settle that exact question.

In this guide we will explain incrementality testing in plain language: what it means, why so much reported marketing "success" can be an illusion, how a proper test is run, and how to use the results to spend smarter. No statistics degree required. By the end you will understand the single most powerful question in marketing measurement: not "how many sales did this get credited with?" but "how many sales would not have happened without it?"

What incrementality actually means

Incrementality is the extra result that a marketing activity genuinely caused, over and above what would have happened anyway. The word that matters is extra. If a campaign is credited with a hundred sales, but ninety of those customers would have bought regardless, then the campaign's true incremental contribution is only ten. The other ninety are what specialists drily call "would-have-happened-anyway" sales, and counting them is one of the most expensive mistakes in marketing.

The baseline you cannot see

The tricky part is that you can never directly observe what would have happened without the campaign, because you did run it. That invisible alternative reality, sales in a world where the campaign never existed, is called the baseline or counterfactual. The whole craft of incrementality testing is about cleverly estimating that invisible baseline so you can subtract it and reveal the true extra effect.

Why attribution alone can fool you

Standard attribution, the system that credits sales back to the touches that preceded them, is genuinely useful, but it has a blind spot. It can only tell you which marketing a buyer was exposed to before purchasing. It cannot tell you whether that exposure changed their decision. Correlation and causation are not the same thing, a distinction we explore in our piece on correlation versus causation, and attribution lives entirely in the world of correlation.

This is why retargeting often looks like a miracle. It deliberately targets people who already showed interest, so naturally many of them go on to buy. The adverts get the credit, but the intent was already there. Without a test, you cannot separate the campaign's real influence from the customers' pre-existing intention, and you may keep pouring money into something that is mostly applauding sales it did not cause. Even a sophisticated approach like data-driven attribution shares this fundamental limit: it shares credit cleverly, but it still cannot prove cause.

How an incrementality test works

The idea is borrowed from medical trials and is beautifully simple. You split your audience into two groups at random. One group, the test group, sees the campaign. The other, the control group, does not. Because the split is random, the two groups are otherwise alike, so any difference in their buying behaviour can be fairly blamed on the one thing that differed: the campaign. The gap between them is the incrementality.

The control group is the magic

That held-back control group is the whole point, and it is also what makes incrementality testing braver than ordinary measurement. You are deliberately not marketing to some people so you can see what they do without you. It feels counterintuitive to spend effort on showing nothing, but that restraint is exactly what reveals the truth. Without a control group you are just guessing at the baseline; with one, you can finally see it. This is the same logic that underpins A/B testing and statistical significance, applied to whole campaigns rather than single web pages.

Reading the result honestly

Once the test runs, you compare the two groups. If the test group bought noticeably more than the control group, the campaign is genuinely creating extra sales, and you can measure exactly how much. If the two groups bought at almost the same rate, that is a hard but valuable truth: the campaign is mostly claiming sales that would have happened anyway, and the money might be better spent elsewhere. Either way, you now know something attribution alone could never tell you.

Why the results often surprise people

The first incrementality test a business runs is frequently a humbling experience. Channels that looked like champions in the attribution report sometimes turn out to add little real lift, while quieter channels prove their worth. This is not a flaw in the test; it is the test doing its job, stripping away the flattering credit that attribution hands out so generously. It reframes how you think about measuring marketing return on investment, because true return is built on incremental sales, not attributed ones.

Getting a test right

A trustworthy test rests on a few non-negotiables. The split must be genuinely random, so the two groups are truly comparable. The groups must be large enough that the result is not just noise, the same sample-size discipline that matters in any experiment. And the test needs to run long enough to capture the full effect, including sales that arrive days after someone first sees the campaign, which connects back to choosing a sensible attribution window.

It also helps to decide in advance what result would change your decision. If you would keep the campaign regardless of the outcome, you do not really need the test. The discipline of committing to act on the answer, before you see it, is what keeps incrementality testing honest and stops it becoming a ritual that simply confirms what you already wanted to believe.

Where incrementality fits in the toolkit

No single method tells the whole truth, and incrementality testing is no exception. It is the gold standard for proving cause, but it is also more effort to run, it requires holding back some audience, and you cannot test everything at once. So the sensible approach is to use it surgically: to settle the big, expensive questions where being wrong would cost the most, and to validate channels that attribution flatters suspiciously well.

Around those tests, you keep using everyday attribution for fast, detailed decisions, and broader modelling for big-picture budget planning. Incrementality is the referee you call in when the stakes are high and the other methods disagree. Watching how a test changes a channel's true contribution can completely reshape your view of your customer acquisition cost, and even how much each genuine new buyer lifts your average order value.

The mindset shift

The deepest value of incrementality testing is not any single result; it is the way it changes how you think. Once you have seen a confident-looking campaign fail to lift sales, you stop taking attribution reports at face value. You start asking the harder, better question about every line of spending: would this sale have happened anyway? That scepticism, applied gently and consistently, is what separates marketing that merely looks busy from marketing that genuinely grows a business.

You do not need to test everything, and you should not. But adopting the incrementality mindset, the habit of always wondering about the invisible baseline, will quietly make every other measurement decision sharper. If you would like help designing a first test around a campaign you are not sure is really working, you are very welcome to get in touch and talk it through.

Frequently asked questions

References

1. Harvard Business Review. "Advertising effectiveness and incrementality analysis." hbr.org.
2. Google. "About conversion lift and incrementality, Ads Help." support.google.com.