In MMM, there is often a dilemma on whether to make model better at explanation or prediction.
Some MMM vendors focus on prediction while compromising on explanation.
But is it the correct approach?
No.
At Aryma Labs, we err on the side of caution and focus more on getting the explanation right first.
𝐓𝐡𝐞 𝐁𝐢𝐚𝐬 / 𝐕𝐚𝐫𝐢𝐚𝐧𝐜𝐞 𝐭𝐫𝐚𝐝𝐞𝐨𝐟𝐟
Most data scientists understand Bias / Variance tradeoff from the lens of overfitting alone.
But if one came from #statistics /econometrics, they would
see the bias/variance tradeoff from the lens of data generating process and estimator bias.
𝐓𝐡𝐞 𝐁𝐢𝐚𝐬:
Bias is generally an attribute of the estimator.
Bias is the difference between estimator’s expected value and the true value of the parameter it is estimating.
An estimator is said to be unbiased if it’s expected value is equal to the parameter that we’re trying to estimate.
𝐔𝐧𝐛𝐢𝐚𝐬𝐞𝐝𝐧𝐞𝐬𝐬 𝐢𝐧 𝐌𝐌𝐌
MMM is all about attribution. There is a true value or ROI of the marketing variable. Through MMM, our job is to hone in on this true ROI.
Our MMM models hence have to be unbiased so that we converge to this true ROI values.
𝐏𝐫𝐞𝐝𝐢𝐜𝐭𝐢𝐯𝐞 𝐚𝐜𝐜𝐮𝐫𝐚𝐜𝐲 ≠ 𝐚𝐜𝐜𝐮𝐫𝐚𝐭𝐞 𝐞𝐱𝐩𝐥𝐚𝐢𝐧𝐚𝐛𝐢𝐥𝐢𝐭𝐲
People often think that MMM models that has low MAPE/ RMSE would be good at explainability of the model too. This is not true. Dr. Galit Shmueli captures this point excellently in her paper ‘To explain or to predict’.
▪ Does good explainability translate to good predictive accuracy?
From our experience, Yes. Models that have less bias automatically capture the data generating process correctly.
If you capture the DGP correctly, you will also have good predictive capability (unless there is a sudden change in the environment or the very characteristics nature of variables included in the model changes).
𝐒𝐨 𝐡𝐨𝐰 𝐝𝐨𝐞𝐬 𝐀𝐫𝐲𝐦𝐚 𝐋𝐚𝐛𝐬 𝐛𝐮𝐢𝐥𝐝𝐬 𝐦𝐨𝐝𝐞𝐥𝐬 𝐰𝐢𝐭𝐡 𝐥𝐞𝐬𝐬 𝐛𝐢𝐚𝐬 ?
▪ We don’t use Bayesian Regression methods.
The priors are the biggest source of bias in your model. As stated in many previous posts, MMM is a small data problem. Your priors will almost always overwhelm the evidence in your data.
▪ We use regularization sparingly
MMM’s multicollinearity problem can be solved via regularization. But that also induces bias. We instead focus on methods like residualization to reduce multicollinearity.
▪ Control for all variables
One of the leading cause of bias in MMM model is not controlling for all variables in the model. Some vendors might say there is no need to include all variable in the models. But such a model will have a lot of bias. Especially Omitted Variable Bias. This leads to endogeneity and hampers causal inference.
𝐈𝐧 𝐒𝐮𝐦𝐦𝐚𝐫𝐲:
MMM is more about causality than prediction. It is always prudent to develop models with less bias.
Resources:
To explain or predict paper :
https://www.stat.berkeley.edu/~aldous/157/Papers/shmueli.pdf
Bayesian MMM vs Frequentist MMM – Key Comparisons
https://www.linkedin.com/posts/venkat-raman-analytics_frequentist-mmm-vs-bayesian-mmm-comparison-activity-7161591828285210624-qyUm?utm_source=share&utm_medium=member_desktop
How we use AIC & KL Divergence in MMM models
https://www.linkedin.com/posts/venkat-raman-analytics_marketingmixmodeling-statistics-datascience-activity-7171760874666254336-Vuv4?utm_source=share&utm_medium=member_desktop
Bayesian MMM’s Stating the obvious problem
https://www.linkedin.com/posts/venkat-raman-analytics_marketingmixmodeling-marketingeffectiveness-activity-7171027667469684736-1j0f?utm_source=share&utm_medium=member_desktop
Which technique provides for great manipulation in MMM – Bayesian or Frequentist?
https://www.linkedin.com/posts/venkat-raman-analytics_marketingmixmodeling-statistics-marketingattribution-activity-7156533790130003968-qVTr?utm_source=share&utm_medium=member_desktop
Adopting MMM for first time ? Use Frequentist MMM.
https://www.linkedin.com/posts/venkat-raman-analytics_marketingmixmodeling-marketingattribution-activity-7148928932862472192-ozrW?utm_source=share&utm_medium=member_desktop
