At Aryma Labs, we constantly endeavor to add as much science as possible to marketing.
MMM model calibration historically has had parallels with multi-linear regression calibration methods.
But MMM is not just linear regression (see link in resources). It has more bells and whistles. As a result, it needs better calibration techniques.
In our MMM calibration process, we innovatively use KL Divergence, Population stability Index and Information theoretic measures (see link in resources).
In addition, we recently started to apply Chebyshev’s inequality as MMM model diagnostic.
📌 Firstly What is Chebyshev’s Inequality?
Simply put, It states that for any random variable with mean μ and a variance σ², the probability of the variable deviating from its mean by more than k standard deviations is at most 1/k².
Chebyshev’s Inequality thus provides an upper bound on the probability that a random variable deviates from its mean by more than a certain amount.
For the mathematically minded, the below is the expression
If X is a random variable with mean μ and variance σ^2, and k is any positive number greater than 1, then the probability that |X – μ| is greater than or equal to kσ is at most 1/k^2.
Mathematically, it can be expressed as:
P(|X – μ| ≥ kσ) ≤ 1/k^2
where:
• P() denotes the probability function.
• X is a random variable.
• μ is the mean of X.
• σ^2 is the variance of X.
• k is a positive number greater than 1.
Note: Chebyshev’s inequality is applicable only if you have finite variance (and there by finite mean). Hence this may not be applicable to fat tail distributions.
📌 How and why we are using Chebyshev’s Inequality in MMM.
Residuals of a model offer clues about goodness of fit but to gauge how good they are, heteroscedasticity plots aren’t enough.
Chebyshev’s inequality provides information about the dispersion of the residuals from the mean. And if a model is poorly fit, we would find data points beyond the upper bound. This is basically our hypothesis.
We had a fun banter with the our client’s data science team and challenged them to build a MMM model alongside us. And then compare our respective model’s fit through Chebyshev’s inequality.
You can guess who won. 😎
📌 The Shiny app
To demonstrate how we applied Chebyshev’s inequality to MMM, we created a shiny app.
As you can see – in our model, no data point is above the bound. But the client model has 7% of the data points beyond the bound.
📌 In summary:
Chebyshev’s Inequality can be valuable metric to gauge MMM model fit.
Resources:
MMM is just linear regression? or is it?
https://open.substack.com/pub/arymalabs/p/marketing-mix-modeling-mmm-is-just?r=2p7455&utm_campaign=post&utm_medium=web
KL Divergence as MMM calibration metric:
https://open.substack.com/pub/arymalabs/p/kl-divergence-as-marketing-mix-model?r=2p7455&utm_campaign=post&utm_medium=web
Assessing relevancy of MMM model through PSI :
https://open.substack.com/pub/arymalabs/p/assessing-relevancy-of-marketing?r=2p7455&utm_campaign=post&utm_medium=web
