After a lot of hard work you have built a great MMM model. The model is now powering saturation / reach frequency curves, scenario planning and Budget optimization.
The client is happy using all of these.
But the client asks you the following question –
“How long can I continue to use the MMM models, When should we update them ?”
At Aryma Labs, we liken MMM models to movie making. Just like a series of snapshots is put together to make a movie, a series of model updates gives you the ‘motion picture’ of your market.
So it is pretty clear that frequent model updates are required. But how to know exactly when to update the model?
The answer lies in Population Stability Index.
๐๐จ๐ฉ๐ฎ๐ฅ๐๐ญ๐ข๐จ๐ง ๐๐ญ๐๐๐ข๐ฅ๐ข๐ญ๐ฒ ๐๐ง๐๐๐ฑ
Population Stability Index (PSI) is a statistic that tells you how much your population (data) has shifted over time or between any interval of time. An excellent paper on PSI by Dr. Bilal Yurdakul can be found in the resources.
PSI is a close cousin to KL Divergence. It is a symmetrised KL Divergence. In my last post (link in resources), I wrote about how we use KL divergence as a MMM calibration metric.
๐๐๐ ๐๐ง๐ ๐๐ ๐๐ข๐ฏ๐๐ซ๐ ๐๐ง๐๐ ๐๐ง๐ฌ๐ฐ๐๐ซ ๐๐ข๐๐๐๐ซ๐๐ง๐ญ ๐ช๐ฎ๐๐ฌ๐ญ๐ข๐จ๐ง๐ฌ.
PSI informs you about data shift.
KL Divergence informs you about model shift.
๐๐จ๐ฐ ๐ฐ๐ ๐ฅ๐๐ฏ๐๐ซ๐๐ ๐ ๐๐๐ ๐ข๐ง ๐๐๐
So once the MMM model is built and put in production (we do this in our product ArymaEdge), we compare the data on which the model was built with the latest data given by the client.
The cadence of providing new data depends on the client. It could be weekly or monthly.
We then compute the PSI between the old data and the new data.
There are some thumb rules as follows:
โช PSI <0.10 : Little shift, no action required
โช PSI between 0.10 – 0.25 : Moderate shift. Investigate data shift, investigate variable shift through CSI (Characteristic Stability Index)
โช PSI > 0.25: Significant shift. Rebuild model
So using these rules, we take a decision on whether the MMM model needs to be updated or not.
๐๐๐ ๐๐ฌ ๐๐๐ ๐ซ๐๐ฅ๐๐ฏ๐๐ง๐๐ฒ ๐ฆ๐๐ญ๐ซ๐ข๐
Marketing environment is always dynamic. One must make sure that the business decisions being made are relevant to current circumstances. PSI helps in evaluating the relevancy of your MMM model.
๐๐ง ๐ฌ๐ฎ๐ฆ๐ฆ๐๐ซ๐ฒ:
Use KL Divergence to assess model fit.
Use PSI to assess MMM model relevancy.
P.S: Image credit in resources.
Resources:
KL Divergence as MMM Calibration Metric
https://www.linkedin.com/posts/venkat-raman-analytics_marketingmixmodeling-statistics-activity-7174277933312724992-X98t?utm_source=share&utm_medium=member_desktop
How often should you update your MMM model?
https://open.substack.com/pub/arymalabs/p/why-should-you-update-your-mmm-model?r=2p7455&utm_campaign=post&utm_medium=web
PSI Resources –
Dr. Bilal’s paper: https://scholarworks.wmich.edu/cgi/viewcontent.cgi?article=4249&context=dissertations
Image credit: https://mwburke.github.io/data%20science/2018/04/29/population-stability-index.html
