Why Synthetic Control Method (SCM) says Yes when the right answer is a Big Fat NO

Honest surgeons don’t advise for surgery when not necessary or when the risk overweighs the benefits.

Honest marketing measurement vendors similarly should ideally tell their clients that sometimes a Geo test is not feasible if a real control market can’t be identified.

But does that happen in practice?
Almost never.

Instead, many marketing measurement vendors choose revenue over rigor.

How do they do it ?
Synthetic Control Method (SCM).

I believe SCM was invented to be a cash cow for the clients.

Here is how it plays out.

Typical Scenario:

Client: “We want to do a Geo test. It is our first time exploring this and hence don’t have any sister market or control market. Can we still do Geo testing?”

Ideal and Truthful Answer: “No. Without a real control, you cannot credibly and causally estimate incrementality”

Greedy Vendor: “Sure, here is a our sophisticated SCM based incrementality tool. Doesn’t matter if you don’t have a real control market. We will create (‘concoct’) one for you.”

Client: Hmmm..okay (I hope I get the right results).

But Results :
Catastrophic. Implementation based on SCM fails miserably.

This failed implementation eventually leads people to have less trust in experimentation overall.

The same happens when a less knowledgeable MMM vendor just wraps his product over some poorly tuned open source model and calls it ‘Futuristic MMM’.

Poor statistical implementation to real marketing problems muddies the water for all genuine practitioners.

People often say “It is not the method that is wrong but the person implementing it”

I disagree, some statistical tools and techniques are indeed flawed by design but people don’t call them out because it enjoys some academic patronage or corporate patronage.

SCM is one such method. Its control market identification can’t be falsified easily, it almost always overfits and gives you false causal estimates.

Not to mention its implementors will hide behind the quagmire of complexity which they purposely construct to first make themselves believe “they are doing some thing sophisticated and scientific” and then to befool the adopters of it.

The uncomfortable truth:
SCM often exists not to solve a real measurement problem but to justify a sale when the correct answer should have been no experiment at all.

I have written extensively about why SCM is fundamentally broken for marketing incrementality.(Links in comments)

Final advice:

If your vendor proposes SCM for Geo testing when no real control market exists: Say no thank you.

You can but thank me later 🙂.