The Ideal Coverage Probability
Coverage probability is a way to evaluate the performance of a confidence interval estimator; ideally, your CI should have the highest possible coverage probability [2]. The usual way to choose a coverage probability is by convention or your best judgment, with 90%, 95%, and 99% being typical choices [3]. However, this isn’t easy: larger prediction intervals (e.g., 95%) might contain all of your values, but these tend to be very wide prediction intervals with little practical value. Setting a too-narrow interval may result in all of your values falling outside the interval, which again is not practical. The goal then, is to find the middle ground.
Coverage Probability vs. Confidence Level
At first glance, coverage probability looks exactly the same as confidence level. However, there are several differences [4]:
- Coverage probability is the probability an interval surrounding the unknown parameter depends on the unknown parameter value; Confidence is the infimum of all coverage probabilities.
- While confidence levels can be calculated by hand, coverage probability is best calculated by computer, as this involves finding the sum of infinite calculations.
- Coverage probabilities tend to be higher than confidence levels if approximations are used to create confidence intervals; Coverage and confidence can be equal when working with continuous distributions (for example, if you’re constructing an interval for the mean of a normally distributed population with a t-distribution [5]); they are never equal when dealing with discrete distributions (for example, when constructing binomial confidence intervals).
References
[1] Lecture 5: Confidence Intervals. Retrieved February 6, 2021 from: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.431.5273&rep=rep1&type=pdf
[2] Harvil, J. (2021). Size and Coverage Probability. Retrieved February 6, 2021 from: https://mediaspace.baylor.edu/media/Size+and+Coverage+Probability/1_u4x1f923/193100023
[3] Landon, J. & Singpurwalla, N. Statistical Practice: Choosing a Coverage Probability for Prediction Intervals.
[4] Cook, P. Coverage vs. Confidence. Retrieved February 6, 2021 from: https://www.mathematica-journal.com/2021/03/03/coverage-versus-confidence/
[5] Meyer, M. (2019). Probability and Mathematical Statistics Theory, Applications, and Practice in R. Society for Industrial and Applied Mathematics.