A lot of people switch to Bayesian methods not because it is better than Frequentist ones, but mainly because they find it hard to wrap their heads around Frequentist concepts.
One such concept is Confidence Interval.
One of the common misconception people have wrt to CI is that “it is the range in which the probability of finding the parameter of interest is 90% or 95%”.
Many conflate this to ‘confidence’ of finding the parameter with a certain probability.
But this is wrong.
Lets first look at the correct definition of CI.
Example : 95% CI.
If one ran the same statistical test taking different samples and constructed a confidence interval each time, then in 95% of the cases, the confidence interval so constructed for that sample will contain the true parameter.
Now that we have got the correct definition, lets answer the question of ‘Confidence’ in confidence Interval.
To understand what the word ‘confidence’ signifies, we need to first consider another word – ‘Coverage’.
Confidence Interval is all about coverage.
That is, if one ran the same statistical test 90 times or 95 times by taking different samples and constructed a confidence interval each time, would they find the parameter of interest in those intervals each time.
To further drive home the point, consider an analogy.
You have a goalkeeper who is static in the ground. Your objective is to erect two poles, each on either side of the goalkeeper (goal post) so as to encompass the goalkeeper.
Even though the goalkeeper is static you find it difficult to place the poles because of fog and your own strange inconsistency in placing the poles around the keeper.
– Sometime you place the poles around the goalkeeper but the poles are too distant from the goalkeeper on either side.
– Sometimes you place it too close to the goalkeeper.
– Sometimes you totally miss the goalkeeper.
In the above example, the goalkeeper is the parameter of interest. If you were to successfully place the goal posts around the goalkeeper 95% of the time, you are said to have 95% CI.
In a way, the confidence is with respect to your skill of placing the goal posts around the goalkeeper each time.
Now you can imagine these goal posts constrict or expand around the goal keeper. This in a way depicts the confidence interval range.
At Aryma Labs, We use Confidence Intervals extensively in Marketing Mix Modeling (MMM), Causal Inference (DiD, RDD etc.) and Experimentations (A/B testing, Incrementality testing etc.).
In each of these cases, the ‘confidence’ connotes to the process of generating these confidence intervals. In an essence it is all about the design of experiment.
Bottomline: The confidence interval is about coverage. The confidence is more to do with the CI generating process ‘the experiments’ rather than the interval itself.
