The Myth of “Triangulation = Truth”

“We used DiD, Synthetic Control and BSTS and triangulated the results.”
“We used MMM, MTA and Experimentation and triangulated the results.”

Both these statements sound very scientific and sophisticated, but they are not.

I have already talked about the second one in my post before (link in comments).

So let me focus on the first one.

To be really honest, I really felt very disappointed that some marketing measurement vendors and ‘causal experts’ celebrated a paper that claims to triangulate between DiD, SCM and BSTS. Which then helped them reallocate $25 Mn !!

Anybody who has a decent knowledge of causality will know, how abhorrent an idea it is to triangulate between DiD, SCM and BSTS.

DiD, SCM and BSTS are not three independent witnesses to the SAME TRUTH.
They are three different ways of making assumptions about the world.

📌 The methods are not interchangeable

Any person who even has a basic understanding of causality would never think of ‘triangulating’ results from DiD, SCM and BSTS.

I mean how can you? I don’t know where to even start.

DiD atleast gives you a causal estimand ATT, but one does not even get any causal estimand from BSTS.

Heck, BSTS is not even a quasi causal experiment. It is just a forecasting algorithm masquerading as ‘causal experiment’.

Also, the name ‘CausalImpact’ should be banned.

SCM – I don’t even want to start. I have written so much about it.
People can read post (linked in comments) and go down the rabbit hole.

But a TL;DR version of SCM is – It concocts a control market by mixing up ‘candidate markets’ through arbitrary and subjective weights. The control market by virtue of design in SCM fits perfectly the treatment market before intervention. But due to this overfitting, there is almost always a ‘divergence’ in the post intervention period.

This is mistaken for ‘causal effect’.

📌 The logical Flaws

One resorts to SCM when they don’t have a control market. If you have already performed DiD, it means that you did have access to a control market. Why then would you resort to SCM?

Similarly why would you resort to BSTS when you already have access to a control market?

Lastly it is ridiculous to even expect that DiD’s ATT = SCM’s ATT = BSTS lift (it does not give you any ATE or ATT or any causal estimand for that matter).

📌 Scientific Marketing Measurement Credibility

I have a lot of ‘Real’ Scientists (physicist, Clinical Trial Experts, Biostatisticians) as connections/friends and they apply causal techniques with a lot of rigor.

I shared the news that in marketing world, we are triangulating between DiD, BSTS and SCM. They just chuckled at the ridiculousness.

If we want Marketing Science to be taken seriously, we need to stop propagating and celebrating incorrect methods.