MMM First? Or Experimentation First?

In our recently conducted Fireside chat (ICYMI – Link in comments), I had asked the panelists:

“Is there an ideal order of Measurement Technique to pursue? What should be carried first – MMM? Experimentation? Causal Experiments?”

The panelists gave really brilliant answers (watch the session to know what they were 🙂).

However in this post, I want to share my two cents on this topic.

📌Start with the eagle’s eye view

I firmly believe that every brand big or small should start with Marketing Mix Modeling (MMM).

Why?

Because MMM gives you the big picture first:

▪️Which channels worked, by how much and in what relative proportion

▪️Good measurement should always move from high level to granular, not the other way around.

MMM (at least done by Aryma Labs) can also give you the granular view courtesy our patent pending GTA-F technique.

Even smaller brands and influencers nowadays have at least two channels (Meta-Google, Meta-TikTok, TikTok-Google etc.).

Hence MMM is the right fit even when there are just 2 channels !!

Don’t believe in the myths that MMM needs more channels or million dollar spends in each. If these are not the restrictions in general linear regression, then it should not be for MMM as well.

📌 Why Not Experimentation First?

Lets suppose you have 2 channels (Google – Meta). Now you do Incrementality tests on these two channels.

But there are couple of statistical problems most analysts overlook: Multiplicity and Family Wise Error Rate (FWER)

Every experiment comes with a false positive risk. When you run multiple experiments, those risks compound.

This is captured by Family Wise Error Rate (FWER). FWER is probability of making at least one false positive across a family of hypothesis tests.

Even if you do 2 experiments within each channel, the FWER comes to about 10% !!

Expand this to 20 experiments, the FWER comes to about 64% !!

For the Geeks formula of FWER is 1-(1-α)^n

So even if nothing actually works, there is a very high chance you will still find a “winner.”

That is why starting with experimentation often creates false confidence.

So you see that a comprehensive and holistic exercise like MMM is better than Experimentations.

📌 When experimentation does come first

If you are experimenting on a new channel for the first time, it is better to do experimentation first. Since chances of this channel showing up in MMM is slim (owing to inconsistent or missing data on previous time periods when the MMM got started).

Also the effect will be very hard to discern given the outsized spends
on other channels.

📌 The Ideal Order of Measurement

1) IMO, the ideal order is MMM first and then causal validation later (through DiD)

2) Experimentation + Causal validation or Causal Experiments itself (DID, ITSA*)
*ITSA is technically not causal though.

Let MMM be the north star in your Marketing Measurement.
Also don’t calibrate MMM through experiments (see why in comments)