There is no uncertainty in MMM
Bayesian Analyst: We prefer Bayesian MMM because it helps us quantify uncertainty.
Me: But there is no uncertainty in marketing attribution.
BA: Pls elaborate.
Me: Sure, you see uncertainty arises when we think the dependent variable’s value is unknowable and it behaves like a random variable.
In case of MMM, we already have the sales numbers (or any KPI for that matter). They are realized and are in the past.
BA: But don’t we also try to find how much each variable contributed to the KPI?
Me: Yes, we do but this is the next step. First we decide which KPI we try to model and next comes the parameter identification i.e. how much each variable contributed to sales.
BA: Ok, you know what, we Bayesians believe there is Uncertainty in parameter identification too.
Me: Yes, I know but that is not logically coherent thought to have.
BA: Explain
Me: We agreed that at the KPI level, there is no uncertainty. Why then there should be a ‘whack-a-mole’ kind of behavior among the independent variables?
Shouldn’t a fix set of numbers result in the fixed number of sales observed?
BA: Hmmm..
Me: Let me elaborate further. In the frequentist world view the parameters are fixed and if you see in MMM, the sales values are fixed and realized. Further only a set fixed number for each marketing variable could have resulted in the ‘fixed’ sales realized in the past.
BA: But we can never be sure about the estimated parameter values. Isn’t it?
Me: Yes, that is true. But that is a drawback or inefficiency on part of the modeling paradigm. The marketing world (especially the past) isn’t uncertain.
To attribute the inefficiency of the modeling paradigm into the real world is wrong.
The frequentist uncertainty is more about the apparatus. The apparatus here being the model. Does the model capture the true parameters or not?
Think of it like catching a static fish in the pond. Each time we throw our net, we waver a little. Hence the net sometimes catches the fish and sometimes it doesn’t.
Ideally, everyone should focus on the apparatus and make sure it is well constructed. That is why Frequentist MMM cares more about Standard Error, P-values and Confidence Intervals.
BA: Ok, but you know what, it is far easier to interpret Bayesian MMM. Example: There is 85% probability that the ROI of Meta is between 3.4 – 5.6.
See how simple that sounds?
Me: Sometimes simplicity ≠utility. Also just to remind that, there is no uncertainty in the real MMM world whatsoever. The above statement assumes uncertainty in the real world. But this statement can be refuted easily in light of decision making.
The range 3.4 – 5.6 does not give confidence. The CMO would ask “what is the ROI exactly? 3.5 or 5.4″. The range is too wide. I need a number to put my money on”
You see quantifying uncertainty has in no way made decisioning any better.
BA: Hmm.. but my manager and others are already sold on Bayesian MMM.
Me: Pls forward this post to them 🙂
