Marketing Mix Modeling (MMM) – Optimize your Marketing ROI

MMMs are long memory models

In my previous posts, I had written about why one should not use Geo tests to fix priors in Bayesian MMM. ICYMI the link is in resources. Another reason why this is such a bad idea is because of the difference in nature of MMM model and Experimentation. What nature? Well, MMMs are long memory models while Experimentation often have no memory or relatively short memory. Let me unpack what I mean by memory (this going to slightly stat/math heavy, but pls bear with me). MMMs are inherently autoregressive in nature. Autoregressive means the present value in some form depends on the past value/s. Autoregressive (AR) Models forecast a time

Calibration is fast, Validation is slow; Calibration does not require sacrifice, Validation requires sacrifice

I have written in my previous posts on why one can only validate (partially) MMMs and not calibrate it through experiments. Let’s quickly recap what is calibration and validation. Calibration is a process where you try to improve the model fit by tweaking various knobs and levers. There are metrics that tell you how well you have calibrated your model. The primary goal of calibration eventually is to reach a ‘Final Model’. Validation on the other hand is a way to test your ‘finalized model’. Simple validation test include testing the model on ‘out of sample data’. But such hold out tests generally provide proof for only prediction accuracy. 📌

Bayesian MMM vs Frequentist MMM – Key Comparisons

One of the fundamental question that you as a client should be asking an MMM vendor is – “Which technique do you employ to build MMM ? Frequentist or Bayesian”. Many vendors (predominantly inclined to Bayesian methods) would often try to dissuade you from going in that direction. Their usual ploy would be to say something along the lines like ▪ Bayesian vs Frequentist – It does not matter. Both are good. ▪ Marketing Effectiveness is not actual science like physics. So cut us some slack. For the first two points, my rebuttal are the following 📌 “Bayesian vs Frequentist – It does not matter. Both are good.” The model

How much data you need for MMM?

Technically the right answer is – It depends. But there are also a lot of heuristics and rule of thumb on how much data is adequate. Statistically speaking we need to have enough degrees of freedom to estimate the parameters. When one has more predictors than data, we call it the P>N problem. In this case, you have more predictors and there isn’t simply enough data to estimate the parameters of these predictors. 📌 The 1 in 10 rule One thumb rule people resort to is the 1 in 10 rule. Basically it says that for every 1 variable we need to have 10 observations. It isn’t clear how or

RCT in MMM violates Principle of Marginality

ICYMI (link in resources), in my previous posts I had highlighted why one can’t RCT MMM. I will just recap two points from that post that will provide a perfect segue for the topic that we are going to cover today. ◾ Marketing Reality is complex and RCT can’t control for all variables When we talk about MMM. It is all about understanding marketing effectiveness i.e. what all variables affect the KPI and what all interactions between them also affect the KPI. One can’t Randomize marketing strategies and neither can one control for all the variables and their interactions. ◾RCTs and Experimentation are uni-variable in nature Most RCTs and Experimentation

What Marketing Mix Modeling domain can learn from Biostatistics

As a statistician, it pains me to see marketers do the following: ▪ Make million dollar marketing decision on just correlation ▪ Specify Marketing Mix Models (MMM) without any statistical rigor ▪ Think that MMM can be specified without controlling for all factors ▪ Think confidence interval from disparate models and experiments can be compared ▪ Think confidence interval talks about some sort of confidence ▪ Poorly understand Randomized control Trials (RCTs) and think that it can be used to validate MMM ▪ Think Beta coefficients of Multi Linear Regression models can be compared to ATE of experiments. ▪ Poorly understand Design of Experiments and thereby conducting wrong experimentation (A/B

Marketing Mix Modeling (MMM) Calibration Experiments

Experimentation has become a buzz word in MMM. Rightly so. Experimentation like Difference in Difference (DID) can help one to holistically prove the efficacy of your MMM model. But, please save yourself the time and money and don’t do RCTs on MMM (check the link in resources to know why). Coming to today’s topic, Experimentation is always considered post hoc (that is once the MMM model is built and finalized). But what if we did experiments just after building a model? In a way to recalibrate the model. Yes, there is a difference between calibration and validation of a model (check the link in resources). So What Experiments can

Maximum Likelihood Estimation and Bootstrapping – The Truth Whisperers in Marketing Mix Modeling (MMM)

The MMM model speaks but the problem is we often don’t listen. Much like a poorly tuned guitar produces noise rather than good notes, your model too shows tell tale signs of poor fit. So who are the truth whisperers? I consider Maximum Likelihood Estimation and Bootstrapping as truth whisperers. 📌 Maximum Likelihood Estimation Simply put maximum likelihood estimation tells you – Given your data, what values of the parameter are more likely to have generated the data. It is simple but very elegant idea. Why is it a truth whisperer? In MMM, our goal is inference and to find the accurate attribution coefficients. MLE as a technique helps you

You can’t RCT Marketing Mix Models (MMMs)

I keep stumbling upon articles and posts where people talk about using Randomized control trial tests (RCTs) to calibrate MMM (absolutely wrong way to go about things – see post link in resources) or to validate the MMM models. In this post we will focus on why one can’t use RCTs to validate the MMM models. RCTs are considered gold standard tests to ascertain causality. The beauty of RCTs technique lies in its name itself – Randomized and Control. Randomized – Through random allocation into test and control one overcomes selection bias. Control – The most important piece of RCT is the control. We can control for the variables that

Why you shouldn’t use media ROI benchmarks to set the priors

We are currently in talks with a company to replace their existing MMM vendor. The company realized that the estimates given by this vendor was consistently inaccurate. We dug deeper and the usual suspects turned up. 1) The vendor was using Bayesian MMM (big big red flag). Ok, since I have already talked so much about why Bayesian MMM is not good. I will be prudent with my keystrokes and instead point you to my previous posts on this topic (link in resources). 2) The second most startling thing the vendor was doing (and topic of my post today) is – setting priors based on ROI benchmarks !! 🖌 Don’t

Blogs