How in-sample MAPE can signal a decline in media effectiveness
In a well built Marketing Mix Model (MMM) or (any time-series model), parameter identification and predictive accuracy usually improve as more data are added.
Temporality wise, an MMM is at its most accurate in the most recent time period than in the beginning. We also published a research paper on the same (link in comments). This is also why for budget optimization one considers the latest 12 months or 6 months data.
But there are cases where the model performance (in-sample MAPE) may start to deteriorate in the most latest time periods.
Now what gives?
When in-sample MAPE starts deteriorating in the most recent months, it can be an early warning of declining media effectiveness.
Let’s unpack what’s happening
Assuming the model is correctly specified:
▪️MMM is expecting media to perform in line with historical response patterns.
▪️But if true effectiveness is declining, the model will systematically over-predict !!
It is not the MMM model which is getting it wrong here.
Ideally, it expects the media to perform to well but because there is a decline in effectiveness, a slight daylight between predicted and actual starts to be seen in the latest time periods.
The above chart is from one of our recent works.
When this chart came up, we immediately felt there is a decline in media effectiveness.
We then checked the ROAS and CPA. We noticed falling ROAS and increasing CPA as time went by.
We also performed a causal experiment DID through our tool DiDective and an ITSA test via AdstockITSA.
Both of them confirmed our results and hypothesis from MMM.
The Media effectiveness indeed started to dwindle in the most recent time periods.
📌 In summary:
✅ Rising recent In Sample MAPE ≠ Bad MMM
It could instead be a leading indicator of declining channel / Media performance !!
Always Validate with:
✅ ROAS / MROAS & CPA trends
✅ Causal tests (DiD) or test like ITSA
✅ Watch MAPE by time segment, not just overall
A single overall MAPE can hide early warning signals.
