We are currently in talks with a company to replace their existing MMM vendor. The company realized that the estimates given by this vendor was consistently inaccurate.
We dug deeper and the usual suspects turned up.
1) The vendor was using Bayesian MMM (big big red flag).
Ok, since I have already talked so much about why Bayesian MMM is not good.
I will be prudent with my keystrokes and instead point you to my previous posts on this topic (link in resources).
2) The second most startling thing the vendor was doing (and topic of my post today) is – setting priors based on ROI benchmarks !!
🖌 Don’t paint everyone with the same brush
As the saying goes, one should also not paint all the brands with the same ROI numbers.
Why?
Even within the same domain, let’s say E-commerce, the ROI benchmark for a platform does not remain static.
Further, more than the ROI of the platform, the ROI is also a function of the creatives.
Because there is always a variance in how each brand designs their creatives/campaigns, the same ROI number is not applicable to them.
If you specify the model with benchmark ROIs, more often than not you will get wrong estimates.
ROI benchmarks at best could be a guide post model fitting but should not be used to specify the model in first place.
The above is one more reason why we prefer Frequentist MMM where one does not have much room to bias the model from the get go.
We are confident that with this client, we will be able to get better accurate estimates with our methods.
Resources:
Which technique provides for great manipulation in MMM – Bayesian or Frequentist?
https://www.linkedin.com/posts/venkat-raman-analytics_marketingmixmodeling-statistics-marketingattribution-activity-7156533790130003968-qVTr?utm_source=share&utm_medium=member_desktop
Want performance guarantees ? choose Frequentist MMM.
https://www.linkedin.com/posts/venkat-raman-analytics_marketingmixmodeling-statistics-marketingattribution-activity-7151460386980945920-H-4U?utm_source=share&utm_medium=member_desktop
Adopting MMM for the first time ? Use Frequentist MMM.
https://www.linkedin.com/posts/venkat-raman-analytics_marketingmixmodeling-marketingattribution-activity-7148928932862472192-ozrW?utm_source=share&utm_medium=member_desktop
Bayesian MMM is not a silver bullet for MMM’s Multicollinearity issue
https://www.linkedin.com/posts/venkat-raman-analytics_marketingmixmodeling-marketingattribution-activity-7106973443807444992-H5Bq?utm_source=share&utm_medium=member_desktop
The Richard McElreath’s Quartet and problem with priors
https://www.linkedin.com/posts/venkat-raman-analytics_marketingmixmodeling-marketingattribution-activity-7107946791983071232-w07F?utm_source=share&utm_medium=member_desktop
