I keep stumbling upon articles and posts where people talk about using Randomized control trial tests (RCTs) to calibrate MMM (absolutely wrong way to go about things – see post link in resources) or to validate the MMM models.
In this post we will focus on why one can’t use RCTs to validate the MMM models.
RCTs are considered gold standard tests to ascertain causality. The beauty of RCTs technique lies in its name itself – Randomized and Control.
Randomized – Through random allocation into test and control one overcomes selection bias.
Control – The most important piece of RCT is the control. We can control for the variables that affect the given phenomenon under study.
Thus the two components allows for causality determination.
Now having understood about RCTs briefly, lets understand why one can’t use RCT to validate MMM models.
◾ Marketing Reality is complex and RCT can’t control for all variables
When we talk about MMM. It is all about understanding marketing effectiveness i.e. what all variables affect the KPI and what all interactions between them also affect the KPI.
One can’t Randomize marketing strategies and neither can one control for all the variables and their interactions.
◾ The window period
Marketing efforts have long term effects. That is why we use econometric techniques to collate data over reasonable large period of time say 52 weeks or 24-36 months for MMM.
This hence allows us to study the long term effects of the Marketing. RCTs and other experiments are short time period exercises.
These seldom capture the long marketing effects. Conducting RCTs or other experimentation for longer time period is not feasible due to cost and resource issues.
◾RCTs and Experimentation are uni-variable in nature
Most RCTs and Experimentation test the effect of one variable on the KPI. In reality Marketing is a multivariable problem.
◾The multiplicity issue
I often hear from proponents of RCTs for MMM that they could technically conduct many experiments like test Instagram ads effect on KPI, TV ads effect on KPI etc.. And they can collate insights from all these.
Well the fundamental problem first is that, none of the experiments are capturing conditional effect of other variables.
Secondly, carrying out multiple tests will increase chances of Type 1 error. MMM in a way is like conducting one single experiment instead of conducting many. Thus mitigating type 1 error.
◾ If not RCT then what? 🤔
For MMM, RCTs are not viable option. Instead one can rely on observational data and quasi causal inference tests like Difference in Difference (see link in resources).
Resources:
Use Experimentation to validate your MMM models, not calibrate it.
https://www.linkedin.com/posts/venkat-raman-analytics_marketingmixmodeling-experimentation-statistics-activity-7155901631820177408-tCs4?utm_source=share&utm_medium=member_desktop
https://arymalabs.com/proving-efficacy-of-mmm-through-difference-in-difference-did/
