When Mathematics Makes Us Forget People: A Blind Spot in Synthetic Control Method (SCM)
Happy New Year Folks,
All of you know I am a stickler for statistical rigor. But over my decade of practice in Marketing measurement, one thing I have realized is – “Don’t forget that all models are just an abstraction of reality”. Mathematics should not make you forget what you are modeling.
One thing I increasingly notice with Synthetic Control Method (SCM) practitioners is how quickly the conversation drifts from markets to mathematics.
Weights, Optimization, Loss functions, Pre period MSPE. Everything becomes beautifully mathematical.
And somewhere along the way, something important gets lost.
A control market is not just set of mathematical equations. It is a market full of real people.
I know SCM analysts will say “there is no real market”. Well, the only way SCM can claim legitimacy and equal footing is to deny the existence of a real control market 😅. Does this remind of you Bayesians? More on this topic later.
Control market comprises of people with:
▪️Different Income distributions
▪️Different Brand Proclivity
▪️Different cultural responses to campaigns
The list goes on and on..
When SCM assigns a weight of 0.37 to Market A and 0.22 to Market B, it creates a mathematically elegant object but that object does not exist in the real world !!
No consumer behaves like “0.37 Dallas + 0.22 Phoenix + 0.41 Tampa”.
The math may be internally consistent, but the interpretation often isn’t.
This is where SCM analysts sometimes get enamored with optimization and forget causal identification.
Minimizing pre treatment error becomes the goal, rather than asking the harder questions:
▪️Would these consumers have behaved similarly absent treatment?
▪️Do these markets really share the characteristics as that of the Treated?
▪️Are we averaging across fundamentally different regimes and markets just to make “lines (curves) line up”?
A synthetic control can look flawless in the pre period and still be causally meaningless if the underlying markets are not structurally comparable.
Causal inference is not a beauty contest for “Fit” metrics.
It is about credible counterfactuals grounded in real human behavior.
The danger isn’t math itself but the danger is letting math subsume you to such an extent that you stop thinking about what underlying process you are modeling and what are the repercussions of implementing those results.
We are running a comprehensive course – Marketing Measurement Marathon where we cover MMM, Causality and Experimentation from first principles starting Jan 5th – 13th.
Check the link in comments to enroll (Last date for enrolling is Jan 4th).
