Coverage Probability in Repeated Sampling

What Coverage Measures

An interval calculated from a sample changes when the sample changes. Coverage probability describes how often the procedure produces an interval that contains the true parameter, when the same sampling process is repeated under the same model. It is a property of the procedure at a specified parameter value, not a property of one observed interval alone.

Definition: Let $X$ denote the sample, and let a procedure report the closed interval $[L(X),U(X)]$, where $L(X)$ and $U(X)$ are random endpoints and $L(X)\le U(X)$. If the sampling distribution of $X$ depends on a parameter $\theta$, the coverage probability of the procedure at $\theta$ is $$ C(\theta)=P_\theta\bigl(L(X)\le \theta\le U(X)\bigr). $$ Here $P_\theta$ means probability calculated under the sampling model with parameter value $\theta$.

To calculate coverage, keep the parameter value fixed and treat the endpoints as random. The central step is to rewrite the event that the interval contains $\theta$ in terms of the sample statistic, then use that statistic’s sampling distribution. Coverage can vary with $\theta$, particularly in discrete models. It is therefore useful to distinguish the general function $C(\theta)$ from a single numerical value at one chosen parameter.

Calculating Coverage From Random Endpoints

The defining event has two requirements: the lower endpoint must not exceed the parameter, and the upper endpoint must not fall below it. A calculation that checks only one endpoint is incomplete unless the model or construction guarantees the other requirement automatically. The following examples show how to use the definition in different sampling models.

Worked Example: A Symmetric Interval for a Normal Mean

Suppose $X_1,\ldots,X_{25}$ are independent observations from $N(\mu,100)$, where the variance $100$ is known. Consider the procedure $[\bar X-2,\bar X+2]$, where $\bar X$ is the sample mean. Find its coverage when $\mu=50$. Name the method and a common mistake it avoids.

Method. Write the interval-inclusion event as inequalities for $\bar X$, then standardize using its sampling distribution. This avoids treating the observed value of $\bar X$ as fixed when evaluating coverage.

The model gives $\bar X\sim N(\mu,100/25)=N(\mu,4)$, so its standard deviation is $2$. The interval covers $\mu$ exactly when $$ \bar X-2\le\mu\le\bar X+2 \quad\Longleftrightarrow\quad \mu-2\le\bar X\le\mu+2. $$

At $\mu=50$, standardizing both limits gives $$ C(50)=P(48\le\bar X\le52) =P\left(\frac{48-50}{2}\le\frac{\bar X-50}{2}\le\frac{52-50}{2}\right) =P(-1\le Z\le1), $$ where $Z\sim N(0,1)$. Thus $C(50)=\Phi(1)-\Phi(-1)=0.8413-0.1587=0.6826$, or $0.6827$ using the more precise normal probabilities, to four decimal places.

Check. The interval has half-width equal to one standard error of $\bar X$, so the result is the standard normal probability within one standard deviation of its mean, about $68.27\%$.

Worked Example: An Interval Based on a Uniform Sample Maximum

Let $X_1,\ldots,X_3$ be independent observations from the continuous uniform distribution on $(0,\theta)$, with $\theta>0$. Write $M=\max(X_1,X_2,X_3)$. A procedure reports $[M,2M]$. Find its coverage for an arbitrary $\theta>0$. Name the method and a common mistake it avoids.

Method. Translate the two endpoint inequalities into conditions on $M$, then use the distribution of the maximum. This avoids confusing the probability that the maximum is below a threshold with the probability required for coverage.

Every observation is below $\theta$, so $M\le\theta$ with probability one. The lower-end condition $M\le\theta$ is therefore satisfied. The upper-end condition is $\theta\le2M$, or $M\ge\theta/2$. Hence $$ C(\theta)=P_\theta(M\ge\theta/2) =1-P_\theta(M<\theta/2). $$

For $0\le m\le\theta$, independence gives $$ P_\theta(M\le m) =P_\theta(X_1\le m,\ X_2\le m,\ X_3\le m) =\left(\frac{m}{\theta}\right)^3. $$ The distribution is continuous, so using $<$ or $\le$ at $m=\theta/2$ gives the same probability. Therefore $$ C(\theta)=1-\left(\frac{\theta/2}{\theta}\right)^3 =1-\left(\frac12\right)^3 =\frac78=0.8750. $$

Check. The interval covers whenever the maximum reaches at least half the upper endpoint of the sampling range. The chance that all three observations fall below $\theta/2$ is $1/8$, leaving coverage $7/8$.

Worked Example: Discrete Coverage for a Binomial Proportion

Suppose $X\sim\operatorname{Binomial}(4,p)$, so $X$ is the number of successes in four independent trials, each with success probability $p$. Let $\hat p=X/4$, and use the interval $[\max(0,\hat p-0.25),\min(1,\hat p+0.25)]$. Find the coverage at $p=0.50$ and at $p=0.25$. Name the method and a common mistake it avoids.

Method. List the integer values of $X$ for which the interval contains the stated $p$, then add their binomial probabilities. This avoids using a continuous approximation for a discrete sampling distribution.

Because each specified $p$ is between 0 and 1, clipping the endpoints to that range does not change whether $p$ is included. The inclusion condition is $$ \left|\frac{X}{4}-p\right|\le0.25. $$ At $p=0.50$, this requires $0.25\le X/4\le0.75$, so $X=1,2,3$. Under $p=0.50$, the probabilities for $X=0,1,2,3,4$ are $1/16,4/16,6/16,4/16,1/16$. Thus $$ C(0.50)=P_{0.50}(X=1,2,3) =\frac{4+6+4}{16} =\frac{14}{16}=0.8750. $$

At $p=0.25$, the condition becomes $0\le X/4\le0.50$, so $X=0,1,2$. The probabilities of the excluded outcomes are $$ P_{0.25}(X=3)=\binom43(0.25)^3(0.75)=\frac{12}{256}, \qquad P_{0.25}(X=4)=(0.25)^4=\frac{1}{256}. $$ Therefore $$ C(0.25)=1-\frac{12+1}{256} =\frac{243}{256} =0.9492 $$ to four decimal places. The same interval rule has different coverage at these two parameter values.

Check. At $p=0.25$, the interval includes the parameter for three of the five possible counts, $X=0,1,2$. Their probabilities sum to $243/256$; counting outcomes alone would not give the coverage.

Why the Endpoint Calculation Works

Coverage is a probability of an event in the sample space. Once $\theta$ is fixed, the event is determined by which sample outcomes make both endpoint inequalities true. Thus the calculation does not require the endpoints to have a particular distributional form: it requires the joint event $L(X)\le\theta\le U(X)$ to be evaluated under $P_\theta$.

For a symmetric interval centered at an estimator $T$, with a fixed positive half-width $h$, the event simplifies algebraically: $$ T-h\le\theta\le T+h \quad\Longleftrightarrow\quad \theta-h\le T\le\theta+h \quad\Longleftrightarrow\quad |T-\theta|\le h. $$ This identity explains why coverage calculations often reduce to a probability statement about the estimator’s sampling error. If the distribution of $T$ is known, the probability can then be evaluated directly, as in the normal-mean example. The simplification is specific to this symmetric fixed-width form; for other endpoints, retain both inequalities until they have been checked.

Fix the parameter value.
Specify the model $P_\theta$ under which repeated samples are generated.

Write the inclusion event.
Start with $L(X)\le\theta\le U(X)$; do not replace it with a statement about only one endpoint unless the other condition is guaranteed.

Translate into sample-statistic conditions.
Rearrange the inequalities carefully, preserving their direction.

Use the sampling distribution.
Evaluate the resulting event using a distribution function, a probability sum, or another justified probability calculation.

Check the result.
Confirm the probability is between 0 and 1 and that the included sample outcomes or tail event match the original interval condition.

Common Mistakes and Their Corrections

The definition is short, but errors often occur when translating it into a probability statement. In the uniform example, the interval $[M,2M]$ covers $\theta$ when $M\ge\theta/2$. A tempting but wrong calculation is $$ P_\theta(M\le\theta/2)=\left(\frac12\right)^3=\frac18 \quad\text{as the coverage.} $$ This is the probability of the complementary event: all observations lie below $\theta/2$, so the interval’s upper endpoint is below $\theta$ and fails to cover. The correction is $$ C(\theta)=P_\theta(M\ge\theta/2) =1-P_\theta(M<\theta/2) =1-\frac18=\frac78. $$

A second trap is to assume that a procedure must have the same coverage at every parameter value. The binomial example gives $0.8750$ at $p=0.50$ and $0.9492$ at $p=0.25$. Because $X$ takes only integer values, small changes in $p$ can change the probabilities assigned to the included outcomes, even when the set of included counts is unchanged. Always identify the parameter value and evaluate the corresponding sampling probabilities.

Key takeaway: Coverage is $P_\theta(L(X)\le\theta\le U(X))$. Fix the parameter, express both endpoint conditions as an event in the sample, and calculate that event under the stated sampling model.

Check Your Understanding

For each problem, identify the sample outcomes or standardized event that produce coverage before calculating the probability.

Let $X_1,\ldots,X_{25}$ be independent $N(\mu,100)$ observations. A procedure reports $[\bar X-4,\bar X+4]$. Find its coverage at $\mu=12$. (Answer: $0.9545$)
For $n=2$ independent observations from the continuous uniform distribution on $(0,\theta)$, let $M$ be the sample maximum and report $[M,1.5M]$. Find the coverage for $\theta>0$. (Answer: $5/9=0.5556$)
Let $X\sim\operatorname{Binomial}(3,p)$, $\hat p=X/3$, and use $[\max(0,\hat p-1/3),\min(1,\hat p+1/3)]$. Find the coverage at $p=1/3$. (Answer: $26/27=0.9630$)
Let $X_1,\ldots,X_{16}$ be independent $N(\mu,64)$ observations. A procedure reports $[\bar X-6,\bar X+6]$. Find its coverage at $\mu=0$. (Answer: $0.9973$)
For $n=4$ independent observations from the continuous uniform distribution on $(0,\theta)$, let $M$ be the maximum and report $[M,1.25M]$. Find the coverage for $\theta>0$. (Answer: $0.5904$)

What You'll Learn