Why Calibrating MMM with Experiments can make you lose trust in both
It goes without saying that Incrementality Testing (Causal Geo tests, Brand Lift tests, and event study) are very powerful.
So are MMMs.
At Aryma Labs, we use both.
Whenever MMMs are not possible, Experimentation can be the answer.
2 years ago, a QSR brand doing MMM with us said – “we launched this special campaign which ran for 2 months. Can MMM tell whether it moved the needle of our sales?”
When all your marketing/media spend variables span 2-3 years and you just have this short duration campaign, MMM is not the answer.
We then looked at Economics for inspiration and took a technique called ‘Interrupted time series analysis – ITSA’ and applied it to our marketing problem.
In another case – a bank wanted us to tell them what all media variables helped them get more home loan leads. As the objective encompassed not one but many media variables effect at play, MMM was the solution.
So you see that both Incrementality testing and MMMs have their strengths and use cases.
📌 The ‘calibrate MMM with experiments problem’
A lot of vendors believe MMM can be calibrated with experiments. We believe it simply can’t be and it shouldn’t be. We even have a research paper with empirical proof that calibrating MMM models with experiments, worsens the MMM almost always.
I particularly like how Venkat Raman explains this in an intuitive way.
“One should not calibrate MMM models with Experiments at all. They model different things, their time horizons are different and so much more.
I think of MMM as a spider web. Each strand (marketing channel) is kind of interconnected with each other. The moment you touch one strand, it sends shockwaves across the whole web. You simply can’t touch one strand and expect nothing to happen to others.
if one is calibrating MMM with experiments, they are essentially trying to just ‘modify’ one strand in web. Because Marketing experiments generally are uni-variable in nature. While your MMM may have TikTok, TV, Meta, Pinterest, Google etc. An incrementality experiment tests only one of them at any time . We can’t take the lift number or ATE from the experiment and try to change the coefficient value in MMM.
That coefficient value in MMM is ‘conditioned upon’ other variables too.”
If you are looking for guidance on how to use MMM and Experiments in tandem (see our articles in comments).
