Why you need much more than R Squared Value to Judge your MMM
Around 2 years ago, a client asked us “Ridhima, I trust you and Aryma Labs but as they say, Trust but verify. Is there a way for us to grade your homework without you having to divulge your secrete sauce?”
That question eventually led to the creation of MMM Diagnose.
📌 The problem with how MMMs are usually evaluated
Traditionally MMMs are judged only on two metrics
– R Squared Value
– Prediction Accuracy
These alone are not enough (you can check the link in comments to know more).
A model can show good fit or good prediction and still be biased, unstable, or incoherent from a business perspective.
We hence spent nearly a year doing a lot of R&D to formulate newer metrics to gauge MMM and we published papers on them too (for the geeky minded, paper link in comments).
📌 What MMM Diagnose Checks Beyond R Squared?
We believe every MMM should pass three categories of checks:
▪️Model Bias Check – This tells you how much your model diverges from the ground truth. We do this check through two tests KL Divergence and Residual Comparison.
▪️Model Coherence Check – Imagine you already built a model in which Google spends was the highest contributor to sales (found to be true through various checks). Now you update the model. The new model tells you Google is not working at all. In such scenarios, it is the model that is wrong. We do this check through Decomp RSSD, NRMSE and PIT tests.
▪️Model Explainability and Prediction Checks
A good MMM model is causal in nature. It should explain the changes in your KPI. R Squared and Adj R squared are good measures to quantify this.
MMM model also predicts well if the model is well fit.
The good prediction should always be a side effect of having fit the model well. We check this through our MAPE check. We further check for over prediction and under prediction. One should be worried more about over prediction than under prediction (link in comments).
📌 Bringing it altogether – The Aryma Score
Looking at many separate diagnostics can itself become confusing.
So we combined nearly a dozen evaluation metrics into a single composite score:
The score is arrived by carefully weighting metrics that are more important.
After extensive testing across real world MMM models, we arrived at a balanced scoring system that helps answer a critical question:
Should you rebuild the model, tweak it, or leave it as is?
📌 What you need to run MMM Diagnose
Running MMM Diagnose requires just four inputs:
– Your Ground Truth KPI (e.g. Sales)
– Your Predicted KPI (from the model)
– How many variables went into the model
– Were any of the variables wrongly signed?
We hit MRR $1000 with MMM Diagnose late last year.
We are glad to have solved one of the pressing problem in MMM – How good is your MMM model really?
You can try MMM Diagnose for free (up to 5 model) – https://lnkd.in/gU5BKw2x
