I know some of you must be wondering what is β-hat and Y-hat.
So lets start this post with a few explainers.
📌 β-hat :
β – hat problems are inference focused. We care about what variables go into the model. What are the parameter values and How much each of the Independent variables affect the dependent variable.
Here the goal is not just prediction but how that prediction was made. Linear Regression’s main goal is retrodiction and not prediction (See my related article in resources).
📌 Y-hat:
Y-hat problems are pure prediction focused. One does not care about what variables go into the model. One does not care about the parameter values.
All one cares about is whether the prediction is accurate or not.
📌 So why a β – hat outlook is better than Y-hat in MMM?
MMM at its core is an inference problem not a prediction problem. This inference also needs to evolve into a causal inference for the MMM to be trustworthy.
However many vendors and analysts alike treat MMM as a prediction problem. I have even seen people apply random forest to do MMM. 🙄
🎯 β – hat outlook helps you calibrate the model better.
Because one cares more about inference, they would also naturally focus more on calibration of the model. And pls don’t conflate calibration with validation. The two are different (Link to my post in resources).
Calibration of a model is about understanding the model fit.
Goodness of fit measures like R squared values, P value, Standard Error, Cross validation and within sample MAPE/MAE/RMSE inform you how well you have fit the model.
Y-hat outlook on the other hand, makes one to overlook calibration and they tend to overtly focus only on validation. But validation can’t happen in silo.
If you see an MMM vendor overtly talk only about validation (or hold out tests), chances are they have been Y-hat focused and have overlooked calibration.
The path to validation is through calibration.
🎯 β – hat outlook helps you to add causality in the model.
With a β – hat outlook, one cares about what variables go into the model or have all the relevant variables in the model. The starting point to achieve causality is – controlling for all the variables that affect the phenomenon at hand. β – hat outlook helps in the regard.
Y-hat focus does not help you bring causality because the question of “how the prediction are made” is never the fundamental question posed. The questions are more centered towards “how accurate my model are?”
🎯 In summary:
In MMM, because the focus is on causal attribution, one would be better served if their outlook is more β – hat. If you specify and calibrate your model well, the positive side effect is that you can also get accurate predictions.
Resources:
Why Linear Regression is not all about predictions – https://bit.ly/48Bylwd
calibration vs validation : https://www.linkedin.com/posts/venkat-raman-analytics_linearregression-statistics-marketingmixmodeling-activity-7132979057750654976-9_03?utm_source=share&utm_medium=member_desktop
β – hat vs Y – hat : https://ryxcommar.com/2019/07/14/on-moving-from-statistics-to-machine-learning-the-final-stage-of-grief/
