Don’t apply Machine Learning Yardsticks to MMM

I had a interesting question from a client last week “Why doesn’t Aryma Labs have a hold out sample for MMM?”

The client was referring to the popular “Train/Test” paradigm.

At Aryma Labs, we have a different philosophy or perhaps the same philosophy that yesteryear statisticians had “Don’t waste data”.

Ideally if your goal is inference, you don’t need to train/test split your data.

In case of prediction, train/test split is justified as the model making such predictions is often black box-ish.

The only way you know your model is working is by testing the predictive
accuracy of the model on an ‘unseen data’.

When the goal is inference, just as it is in case of Marketing Mix Modeling (MMM), train/test split means that your are ‘wasting data’.

The 25% or 30% data that could have been utilized for better specification of the model and thereby better understanding of data generating process is unnecessarily wasted to check the predictive power of the model.

MMM being a variant of Linear Regression is more about inference rather than prediction (check the link in comments).

The fact that MMM could be used for prediction is just a positive side effect of having specified the model properly. The prediction is just a special case of retrodiction.

📌 The practical problem of 80-20 or 70-30 splitting of data.

When we split the data into 80-20 (training – test), we almost always sacrifice the most recent 20% of our data.

In our research paper we show that the MMM model is the most accurate and stable in its most recent months (see link in comments).

Such being the case, why would you want to waste the most accurate period of your data?

▪️ The budget optimization

The budget optimization is carried using the most recent data (either last 1 year or last 24-52 weeks).

Why?

– It reflects current response curves.
– It aligns with current strategy and execution.

If you exclude this data from model estimation, you are weakening the very foundation on which decisions are made.

📌 So how should you test your MMM model?

To check the MMM’s performance, one could utilize Cross validation.

In an essence Cross Validation tests the model on the entire dataset although in subsequent sub samples or folds.

One kind of gets an ‘average’ measure of model consistency via CV.

Where as in Train-Test, not much can be known about model consistency merely based on one test set.

The only caveat wrt to CV in case of MMM is that one must be circumspect of the temporal effects. MMM has a temporal component and hence one should not shuffle the data haphazardly and perform CV.

This is akin to why one does not do shuffled CV in case of time series forecasting.

The right way to perform CV would be sequential, i.e. respecting the temporal aspect.

So:
✅ If your goal is inference, use all the data.
✅ If your goal is prediction, test on unseen data.