Why “Test vs Total” Geo Testing is Misleading and Not Causal.

We onboarded a client early this month and we were shocked to find what their earlier experimentation vendor had implemented for them.

I always knew SCM experiments are no good and similarly most SCM vendors always try to pull a wool over the eyes.

Here is what happened:

The client was experimenting running Meta and Google Ads in 10 states in USA. The client was approached by this experimentation vendor and was even given free 6 months trial of their so called sophisticated SCM tool.

The client wanted to carry out causal experiments and asked the vendor if they can do it? The client also mentioned that they didn’t have any control markets.

Now here is what the vendor did and I am sure every causal inference analyst after hearing their suggestion will go like “What the hell?”

The vendor told client “Don’t worry if you got no control markets.” We will pick 2 or 3 markets as the test and for the control – it will be the total of all the markets (which includes the test markets) 🤡.

I have written a lot about how synthetic controls are bad. But this is next level ridiculous.

Lets unpack why the above is wrong:

📌 Contaminated Control

The control in the above case is : Total markets (which includes the test states too !!)

So we are literally comparing a subset to a superset that already contains it.

Any lift in test markets is partially injected back into control markets.

📌 The Muted Effect Scenario

What we are estimating is Lift = Test Markets – (Test Markets + Non test markets).

You can guess that this will tend towards to zero. Regardless of the true lift at play.

This is the muted effect.

The client had complained that almost all their tests came out as “no lift” detected. Well one can see why.

📌 No counterfactual.

The core tenet of causality is counterfactual.

A true causal design would have answered “what would have these 3 or 4 test markets done without the treatment (ad campaigns)?”

But the vendor’s action changed the question to “What did these 3 or 4 test markets do relative to themselves + everyone else”

This is in no way a counterfactual test.

📌 The SUTVA violations

The vendor technically included the treatment markets into the control. But in reality, a control is the one which absolutely receives no treatment. In this case ideally, it should not have seen any ad campaigns. But we know that is not the case here.

Eventually the above test does not measure “Did the business grow because of this ad spend?”
It rather measures “Did the test states grow faster than the rest?”

A proper causal experiment has the following:

▪️Treatment group
▪️Untreated control group
▪️Clear Estimand: E[Y(1)−Y(0)]

And most importantly, never ever make the treatment markets the control markets.

When you lack real control markets, a lot of shenanigans like above is possible. As clients you need to be wary of this.