Prior Predictive Check – Checking your Assumptions against your Assumptions !!

A Prior Predictive Check (PPC) is often presented in Bayesian circles as a kind of “sanity test before modeling”.

But what it actually does and what people think it does are two very different things.

📌 What is a Prior Predictive Check?

Before seeing any data:

▪️You specify priors for your parameters
(say in MMM parlance they would be ROI numbers or contribution numbers etc.)

▪️You then Sample parameters only from the priors and push those sampled parameters through your model to simulate outcomes.

So far so good.

But here comes the catch – how do you validate that the data from this process is accurate and trustworthy?

The short answer – you don’t (but Bayesians believe they do) 🙂.

Yesterday one commenter opined “You simulate from your priors before fitting and verify they produce plausible data. If they don’t, you fix them before the prior ever touches the model.”

If you read carefully you can easily find the flaw with this approach.

Firstly the question begs – Plausible in terms of what? Historical ROI range? channel contribution expectations?

Bayesians in this approach are still stuck inside their bubble of ‘plausibility’. This doesn’t have touch with reality whatsoever.

You are essentially checking your assumptions against your assumptions.

📌 Why This Can Be Misleading (Especially in MMM)

The above can be summarized as

“If my beliefs about media effectiveness (ROIs) are true,
do the outcomes generated from my beliefs look believable to me? 😅”

This is circular with absolutely no touch with reality here.

📌 Encourages Outcome Engineering Before Data

In practice, I have seen analysts do the following:

Simulated ROI too high? -> tighten priors

Simulated contribution number too small ? -> widen priors

Eventually, one arrives at priors that:

– Produce ‘realistic’ sales / ROIs
– Don’t offend domain intuition

But you have engineered the model based on desired outcomes and not falsifiable evidence. This is one of the reasons people say Bayesian approaches are unfalsifiable.

📌 Plausible Data ≠ Ground Truth

Another example

True channel ROI = 0.9
But your belief = 0.2 – 0.5

Your original prior generates simulated worlds with large ROIs. But you disagree and tighten the priors even more.

Now post PPC, your simulated worlds look “reasonable”. But what you have done is, prevent the model from ever discovering the true ROI (0.9) before seeing any data !!

So a PPC passing model may still:

❌ Misattribute
❌ Suppress base (a common complaint in Bayesian MMM)
❌ Show stable ROIs

and you won’t know until it’s too late.

That’s why at Aryma Labs, we build our MMM models on falsifiable Frequentist principles.

We believe in Falsifiability for Experimentation too. That is why we place more trust in Difference-in-Differences (DiD) than SCM.

By the way SCM is causal world’s Bayesian.😅
Read the post (link in comments).