Why “Peeking” in Experiments Can Break Your MMM
In our recent Marketing Mix – Unmixed podcast with Premjeet Sodhi, Premjeet mentioned a very interesting point.
“The beauty of MMM is that it is non intrusive. You can let the world happen as it is, You can let life happen and then you can pick up the data about what measures it and gain insight into it”.
An excellent point. While MMM is non intrusive, a lot of things can intrude in it and perhaps make MMM infeasible !!
One such intrusion is Peeking.
📌 What is Peeking?
In Experimentation, Peeking is the act of taking a a peek at your experiment and stopping the experimentation prematurely just because at the time of ‘peeking’ statistical significance has been attained. Basically one stops the experiment without it running its full course.
📌 How Peeking can hamper your future MMMs?
When an analyst peeks early and stops the experiments abruptly by:
• Pausing a media campaign mid flight
• Restarting it weeks later
• or turning spend on – off multiple times
they unintentionally create structural missingness and discontinuities in the media data.
This data when fed into the MMM can causse a lot of issues. From rendering the whole MMM infeasible to mispecifying the MMM.
MMM relies on stable exposure patterns to:
• Learn adstock
• Separate media effects from seasonality and noise
• Estimate saturation curves reliably
Abrupt start – stop behaviour does the opposite. It causes the following:
• Carryover gets truncated artificially
• Coefficients become unstable
As a result, your ROAS, MROAS numbers becomes unreliable and as a further domino effect, your budget optimization and media planning to gets hampered.
An experiment that was peeked at and stopped early may feel “rigorous” in isolation, but once that same media enters an MMM, it degrades the entire system.
Bad experimentation practice today becomes bad MMM tomorrow !!
If you plan to run experiments and include those campaigns in your MMM, please keep in mind that bad experiment data can really harm your MMM.
Experiment design cannot be siloed from MMM requirements.
We at Aryma Labs consult both on experimentation and MMM for many companies. We make it a point to forewarn clients about the perils of peeking.
You can peek all you want but don’t act on it 😅
