While confidence intervals are fundamental tools in statistics, their interpretation and application are not always straightforward. The concept of a confidence level, often set at 95%, refers to the long-run frequency with which a procedure for constructing intervals will capture the true parameter. However, this nominal coverage probability can sometimes diverge from the actual coverage probability, leading to potential misinterpretations. Understanding
these nuances is crucial for accurate statistical inference and avoiding naive conclusions.
Challenging Naive Interpretations: Welch's Example
Welch presented a notable example that highlights the difference between the theory of confidence intervals and other interval estimation theories, such as Fisher's fiducial intervals and objective Bayesian intervals. This example, considered a well-known counterexample for Neyman's version of confidence interval theory, demonstrates that even an optimal confidence procedure can exhibit counter-intuitive properties. For instance, in a specific scenario involving independent observations from a uniform distribution, an optimal 50% confidence procedure was shown to guarantee 100% coverage when the observed data points were far apart, yet almost 0% coverage when they were very close together. Despite these extremes, the procedure still achieved 50% coverage on average.
The Disconnect Between Nominal and Actual Coverage
This example underscores a critical point: the nominal confidence coefficient, such as 50%, may not directly relate to the uncertainty one should have that a *specific* interval contains the true value. The two counter-intuitive properties—100% coverage in some cases and near 0% in others—balance out to yield the nominal 50% coverage on average. However, this means that even an optimal procedure might not provide an assessment of the precision of the estimate or the uncertainty that a particular interval contains the true value. This situation serves as a strong argument against naive interpretations of confidence intervals, emphasizing that any asserted properties beyond nominal coverage must be rigorously proven.
When Assumptions Fall Short: Discrete Distributions
A common source of discrepancy between nominal and actual coverage probability occurs when a discrete distribution is approximated by a continuous one. The construction of binomial confidence intervals is a classic illustration where coverage probabilities rarely precisely match nominal levels. In such cases, if the assumptions used in deriving a confidence interval are not fully met, the actual coverage probability can be either less than or greater than the nominal coverage. An interval is termed "conservative" if its actual coverage probability is greater than the nominal, and "anti-conservative" or "permissive" if it is less. This highlights the importance of carefully considering the underlying assumptions and the nature of the data when interpreting confidence intervals, especially for discrete outcomes.
The Broader Concept of Confidence Distribution
Beyond the specific confidence interval, the concept of a confidence distribution (CD) offers a more comprehensive framework. A CD is a function on the parameter space that can represent confidence intervals of all levels for a parameter of interest. While distinct from a probability distribution function of the parameter, it is a purely frequentist concept that can provide a wealth of information for constructing various frequentist inferences, including point estimates, confidence intervals, critical values, statistical power, and p-values. This broader perspective acknowledges the complexities and varied interpretations that arise when moving from a single interval to a more complete representation of confidence across all possible levels.
