How Train – Test Validation Quietly Changes the Philosophy of MMM

Yesterday, I interviewed a candidate for an MMM role from a rival MMM company. Initially, I was genuinely impressed.

They spoke about a fairly rigorous frequentist setup and mentioned several checks beyond just R squared to evaluate model quality. That was refreshing to hear because MMM quality cannot be judged by R squared alone.

But then the candidate mentioned that they perform the exact same checks on the test dataset as well !!

I was a bit taken aback.

I further asked what they would do if the prediction accuracy went down in the test data (which almost always happens). They said they try to include new variables or tweak the parameters till they get a good test data prediction accuracy.

This is where all the problem starts – making prediction accuracy the objective function.

The moment you split MMM into Train and Test sets, you inevitably start becoming prediction focused.

And that changes the philosophy of the model itself.

This is one of the reasons why many MMMs slowly drift away from attribution and become forecasting exercises disguised as attribution systems.

📌 Why does this happen?

The moment you create: Train Set for Fitting Model and Test Set for evaluating Prediction Accuracy, you introduce a new optimization pressure: “How do I maximize out of sample prediction?”

And once that pressure enters the system, the model naturally starts favoring variables that improve predictive fit, even if they are not causally meaningful.

📌 Prediction ≠ Attribution

I have written extensively on why MMM is not just about prediction (Link in comments).
The fact that any MMM predicts well should be a side effect of having fit a good causal model in first place.

Any vendor telling you this is just looking for the easy way out and wants to escape culpability. Prediction accuracy is easy to show, also easy to grasp by people.
The optics is simpler and appealing. But accurate attribution is not necessarily manifested only through prediction.

📌 Prediction Accuracy Focus – The slippery slope

Once holdout metrics become the north star, the following happens:

– Teams start optimizing for MAPE
– Vendors start showcasing prediction accuracy
– Complex temporal smoothing gets rewarded
– Media decomposition quality becomes secondary

Slowly the MMM becomes: “Can I predict next week?”

Instead of: “What truly drove last week?”

Also paradoxically, you can’t predict next week well in MMM unless you have accurately figured out what factors lead to the change in the KPI last week.

Causality is the cheat code for accurate predictions.

A good attribution model may forecast well. But a good forecasting model is not automatically a good attribution model.

That distinction matters enormously when millions of dollars in budget allocation depend on the decomposition.

A Good MMM model does not predict the future. It helps in changing the future.