Nothing Credible about Credible Intervals and Why this matters when reading ROAS or iROAS from Bayesian MMM
Credebile : Able to be believed or capable of persuading
Many people think Credible intervals are equivalent to confidence interval. They are not.
Frequentist methods focus on the apparatus or the experiments. Every commentary like p-value and confidence interval is actually a commentary about the apparatus and not the phenomenon.
The phenomenon is assumed to be ‘there’ – ‘fixed’ and ‘true’. The question then boils down to – is apparatus good enough to capture this phenomenon (the parameter).
The question that frequentist confidence Interval answers is that – if the parameter is found, true and fixed ; would our experimental set up be able capture this parameter?.
Now because Frequentists draw ‘confidence’ from long run frequencies. If we perform 100 experiments and lets say 95 of the times we are able to capture the true parameter, then we are said to have 95% confidence interval.
You see the word ‘confidence’ is actually the confidence of coverage.
The coverage that if I performed my experiments 100 times, I would be able to hone in on the true parameter 95 times.
On the other hand, Bayesian methods try to capture the ‘varying’ phenomenon. They treat parameters to be a random variable (even though this is a wrong supposition most of the time. It is certainly wrong in case of MMM).
The uncertainty that Bayesians try to ‘quantify’ is actually a deficiency of the modeling paradigm itself.
📌 The missing credibility in credible Intervals
The credible intervals are actually a misnomer. I mean if you yourself are unsure and say that there is uncertainty, then you tell somebody
CI of ROAS is 90% CI: 2.24 – 3.40, you are implying that there is still 10% probability that the ROAS value is out of this range.
When you are in the probability realm, how can you talk about persuasion when you yourself are not ‘persuaded’ enough 🙂 ?
📌 The problem with using Bayesian credible Intervals in MMM
Firstly, if you use Bayesian MMM, you could be reporting a very inflated ROI numbers (up to 2x). Check the article link in comments.
Because CI is actually a range, Marketers demand a precise number.
To give a precise number Bayesians either resort to giving the mean, median, mode or sometimes HDI.
In MMM, Bayesians don’t use normal distribution to specify priors.
Bayesians use half normal distribution or log normal distribution because they want to avoid the problem of getting negative values.
Many marketers advice you to take the median but once you are in the log normal land, the median is also an inflated number as compared to the HDI.
Picking median, mode or mean – none of it solves the problem because the log normal is skewed. (See the image).
If you are a client, don’t get carried away by the high sounding Bayesian words. Bayesian methods purport to solve many problems but in reality it only exacerbates them.
