Interpreting Weighted Confidence Intervals

What a Weighted Confidence Interval Describes

A weighted confidence interval is a way to express uncertainty about a population quantity estimated from a probability sample. Its interpretation depends on keeping two things separate: the population quantity being estimated and the random process that selected the sample. The interval describes uncertainty caused by sampling under the survey design; it does not describe the range of values among people in the population.

For a finite-population interpretation, imagine a defined population of \(N\) units, with a value \(y_i\) attached to each unit \(i\). A target such as the population mean or proportion is fixed once the population values are fixed. The sample is random because the survey design assigns probabilities to possible samples. As a result, an estimate—and an interval calculated from that estimate—can vary from one sample to another even though the target does not.

The target might be a population mean, proportion, total, quantile, or value of a distribution function. A weighted point estimate uses the survey information and weights to estimate that target; a design-based standard error represents the estimate’s sampling variability under the design. The interval combines the estimate and its standard error. It is the target that gives the interval its meaning: a confidence interval for a population mean is not an interval containing most individual observations.

Repeated-Sampling Interpretation

Suppose the population and its target \(\theta_U\) are fixed, and let \(\hat{\theta}\) be a survey-weighted estimator. A common large-sample interval has the form

where \(SE(\hat{\theta})\) is a standard error that accounts for the sampling design and \(c\) is an appropriate critical value. For an approximately normal procedure, \(c=1.96\) gives a nominal 95% interval. Other procedures may use a design-based degrees-of-freedom adjustment or a different interval construction; the critical value should match the procedure and its assumptions.

The repeated-sampling statement concerns the whole procedure. If the same population were sampled many times using the same design, each sample would produce its own estimate, standard error, and interval. The target would remain fixed. The confidence level describes the long-run proportion of those intervals that contain it; it does not assign a probability to the target moving into or out of the particular interval from the sample already observed.

This interpretation requires more than calculating an estimate and multiplying its standard error by a constant. The sampling design and target must be defined; the variance estimator must reflect the relevant design features; and the interval approximation must be adequate. For an asymptotic normal interval, the estimator’s design-based sampling distribution should be sufficiently close to normal, and the estimated standard error should be sufficiently reliable. Sparse data, boundary estimates, or too little information for variance estimation can undermine those conditions. A nominal confidence level is not a guarantee of actual coverage in every sample or under every design.

Interval Uncertainty Is Not Population Variation

A population distribution describes how values differ across units. For example, the population standard deviation describes the spread of individual incomes around the population mean. A confidence interval’s width instead reflects the uncertainty of an estimator under repeated samples, as represented by its standard error and the chosen interval procedure. These quantities have different meanings and generally different units of calculation: the standard deviation concerns individual values, while the standard error concerns an estimate.

A population can have wide variation among individuals and still yield a relatively precise estimate of its mean if the survey design supplies substantial information. Conversely, an estimate can be uncertain even when individual outcomes are not very spread out, for example if the sample provides little information about the target. Survey design features affect precision, so the interval should be interpreted using the standard error appropriate to the design rather than an assumption that observations were obtained by simple random sampling.

The same distinction applies to proportions and quantiles. A confidence interval for a population proportion concerns uncertainty about the proportion, not the range of binary outcomes for individuals. An interval for a population quantile concerns uncertainty about the location of that quantile, not an interval expected to contain a stated fraction of individual values. The latter question concerns the population distribution itself.

Confidence Intervals Are Not Prediction Intervals

A confidence interval for a population quantity and an individual-level prediction interval answer different questions. A confidence interval for a mean addresses uncertainty about the mean of the defined population. A prediction interval would address the possible value of a future or otherwise specified individual outcome under an appropriate model. It must account for individual-level variation as well as any uncertainty in the estimated quantity or model. A confidence interval for the mean does not automatically provide that individual-level range.

For example, an interval around a weighted mean completion time is not a range expected to contain 95% of people’s completion times. To describe where individual times lie, one might summarize the population distribution or estimate specified quantiles. To make a prediction about a new individual, a prediction method and its assumptions would be needed. The fact that both kinds of intervals may be written with lower and upper endpoints does not make their interpretations interchangeable.

Why the Repeated-Sampling Argument Works

The basic justification for a normal-based interval is that, under suitable design-based regularity conditions, the standardized estimation error is approximately standard normal. If the standard error estimate is reliable, then

For a standard normal random variable \(Z\), \(P(-1.96\leq Z\leq 1.96)\) is approximately 0.95. Applying this central probability to the standardized estimation error gives

Rearranging the inequalities gives

This is a repeated-sampling coverage statement: the endpoints are random across samples, while \(\theta_U\) is fixed. It is approximate because the standardized error may not be exactly normal and the standard error is estimated. The justification therefore depends on the design, the estimator, the variance procedure, and the adequacy of the approximation—not just on the number 1.96.

Common Pitfall

Incorrect reasoning: “There is a 95% chance that the population mean is inside the interval, and 95% of individuals are inside it.” These are two distinct errors. In the finite-population design-based interpretation, the parameter is fixed rather than random, and a confidence interval does not describe the distribution of individual values.

Correction: State the target and the repeated-sampling procedure. For example: “This procedure is designed to produce intervals containing the region’s fixed population mean in approximately 95% of repeated samples drawn under the specified survey design.” If the question concerns individual values, use a suitable description of the population distribution; if it concerns an individual prediction, use a method designed for that purpose.