Conditional Coverage Given the Design Matrix

Why Condition on the Design?

In regression, uncertainty in an estimate depends on the values in the design matrix. A confidence interval or prediction interval calibrated from a normal linear model is not merely accurate on average across possible designs: under the model’s assumptions, it has its stated finite-sample coverage for each fixed full-rank design matrix. This is the meaning of exact conditional coverage.

The qualification matters. The guarantee is exact when the errors, conditional on the design, have the specified normal distribution and the design is fixed independently of those errors. It need not survive a data-dependent selection of the design or model that changes the conditional error distribution.

Definition: Let $X$ be a fixed $n\times p$ design matrix of rank $p$, and consider the normal linear model $Y=X\beta+\varepsilon$, where $\beta$ is fixed and $\varepsilon\sim N(0,\sigma^2I_n)$, with $\sigma^2>0$. A confidence procedure for a target has exact conditional coverage $1-\alpha$ if, for every such fixed $X$, its probability of containing the target, calculated under the model’s sampling distribution, is $1-\alpha$. If $X$ is random, this means the corresponding probability conditional on $X$ is $1-\alpha$, almost surely.

Write $G=X'X$, which is invertible because $X$ has rank $p$. The least-squares estimator and residual variance estimator are

\hat{\beta}=G^{-1}X'Y,\qquad s^2=\frac{(Y-X\hat{\beta})'(Y-X\hat{\beta})}{\nu}, \qquad \nu=n-p.

We require $\nu>0$. The matrix $X$ is treated as fixed in the sampling distribution. When $X$ is random but independent of the errors and the model holds conditional on $X$, the same calculations apply for each realized full-rank $X$.

Why the Scalar Confidence Interval Has Exact Coverage

For any nonzero vector $c\in\mathbb{R}^p$, the target $c'\beta$ is a linear contrast. Under the model, $\hat{\beta}$ is normal with mean $\beta$ and covariance matrix $\sigma^2G^{-1}$. Meanwhile, the residual sum of squares estimates the same error variance. The crucial fact is that the estimator and the residual variance estimator are independent under normal errors.

Theorem: Under the stated model, with $X$ fixed and full rank and $\nu=n-p>0$, $$ \frac{c'\hat{\beta}-c'\beta}{s\sqrt{c'G^{-1}c}}\sim t_{\nu}. $$ Consequently, if $t_{\nu,1-\alpha/2}$ is the $1-\alpha/2$ quantile of the $t_\nu$ distribution, then $$ \left[ c'\hat{\beta}-t_{\nu,1-\alpha/2}s\sqrt{c'G^{-1}c},\ c'\hat{\beta}+t_{\nu,1-\alpha/2}s\sqrt{c'G^{-1}c} \right] $$ has coverage exactly $1-\alpha$ for $c'\beta$, conditional on $X$.

To prove the result, let $P_X=XG^{-1}X'$ be the orthogonal projection onto the column space of $X$, and let $M_X=I_n-P_X$. Then $\hat{\beta}-\beta=G^{-1}X'\varepsilon$, so

\frac{c'\hat{\beta}-c'\beta}{\sigma\sqrt{c'G^{-1}c}}\sim N(0,1).

Also, the residual vector is $M_XY=M_X\varepsilon$. Since $M_X$ is a symmetric idempotent matrix of rank $\nu$, the normal projection result gives

\frac{\nu s^2}{\sigma^2} =\frac{\varepsilon'M_X\varepsilon}{\sigma^2} \sim \chi^2_{\nu}.

The numerator $c'G^{-1}X'\varepsilon$ is a linear function of $P_X\varepsilon$; the residual vector is a linear function of $M_X\varepsilon$. The two projections are orthogonal, since $P_XM_X=0$. They are jointly normal, so their zero cross-covariance implies independence. Thus the displayed standard normal variable is independent of $\nu s^2/\sigma^2$. By the definition of the $t_\nu$ distribution, their ratio, with denominator $\sqrt{(\nu s^2/\sigma^2)/\nu}=s/\sigma$, has a $t_\nu$ distribution. The interval contains $c'\beta$ precisely when the absolute value of this statistic does not exceed $t_{\nu,1-\alpha/2}$, an event with probability $1-\alpha$. All probabilities here are under the fixed $X$ sampling distribution. $\square$

Worked Example: A Slope Confidence Interval at Fixed Design

Suppose $n=8$ observations follow the normal linear model with an intercept and a slope. The observed predictor values are $-3,-2,-1,0,0,1,2,3$. Thus $p=2$, $\nu=8-2=6$, and

X'X= \begin{pmatrix} 8&0\\ 0&28 \end{pmatrix}.

The off-diagonal entry is zero because the predictor values sum to zero; the lower-right entry is their sum of squares, $28$. Suppose the fitted slope is $\hat{\beta}_1=0.40$ and $s=2$. For the slope, $c=(0,1)'$, so $c'(X'X)^{-1}c=1/28$. The standard error is therefore

s\sqrt{c'(X'X)^{-1}c} =2\sqrt{\frac{1}{28}} \approx 0.378.

For a two-sided 95% interval, $t_{6,0.975}\approx2.447$. The margin of error is $2.447(0.378)\approx0.925$, giving

0.40\pm0.925=[-0.525,\ 1.325].

This interval’s 95% coverage is exact for this fixed design under the model assumptions. It does not mean there is a 95% probability that the fixed slope lies in this particular realized interval; rather, the interval-producing procedure covers the slope in 95% of repeated samples at this design.

Prediction Coverage Includes a New Error

A confidence interval for a mean response concerns a fixed quantity. A prediction interval for one future response concerns a random outcome and must include the future observation’s error. Suppose the future response at a fixed design vector $x_0\in\mathbb{R}^p$ is

Y_0=x_0'\beta+\varepsilon_0,

where $\varepsilon_0\sim N(0,\sigma^2)$ is independent of the training errors. The prediction error $Y_0-x_0'\hat{\beta}$ is then normal with mean zero and variance $\sigma^2\{1+x_0'G^{-1}x_0\}$. The $1$ accounts for the new error; $x_0'G^{-1}x_0$ accounts for estimating the mean response.

Theorem: Under the fixed-design model above, if the future error is independent and $N(0,\sigma^2)$, then $$ \frac{Y_0-x_0'\hat{\beta}} {s\sqrt{1+x_0'G^{-1}x_0}}\sim t_{\nu}. $$ Thus the prediction interval centered at $x_0'\hat{\beta}$, with half-width $t_{\nu,1-\alpha/2}s\sqrt{1+x_0'G^{-1}x_0}$, covers $Y_0$ with probability exactly $1-\alpha$, conditional on $X$.

Indeed, $Y_0-x_0'\hat{\beta}=\varepsilon_0-x_0'G^{-1}X'\varepsilon$. Its two terms are independent normals, so its variance is

\sigma^2+\sigma^2x_0'G^{-1}X'XG^{-1}x_0 =\sigma^2(1+x_0'G^{-1}x_0).

This prediction error is independent of the residual sum of squares: it depends only on the future error and the projection of the training errors onto the column space of $X$, whereas the residual sum of squares depends on the orthogonal residual projection. The same independent chi-square denominator used for the contrast interval therefore gives the stated $t_\nu$ pivot. The interval covers exactly when the absolute value of the pivot is at most $t_{\nu,1-\alpha/2}$, which has probability $1-\alpha$. $\square$

Worked Example: Predicting One Future Response

Use the design and fitted values from the preceding example, and consider a future observation at $x_0=(1,2)'$, corresponding to predictor value $2$. Suppose $\hat{\beta}_0=5$, $\hat{\beta}_1=0.40$, and $s=2$. The predicted response is

x_0'\hat{\beta}=5+2(0.40)=5.8.

Here $G^{-1}$ has diagonal entries $1/8$ and $1/28$, so

1+x_0'G^{-1}x_0 =1+\frac18+\frac{2^2}{28} =\frac{71}{56}.

The prediction standard error is $2\sqrt{71/56}\approx2.252$. With $t_{6,0.975}\approx2.447$, the margin is approximately $5.51$, and the 95% prediction interval is

5.8\pm 5.51=[0.29,\ 11.31].

Its exact coverage concerns a future random response, not the fixed mean $x_0'\beta$. Omitting the $1$ from the variance factor would instead calibrate uncertainty about the mean and would generally understate prediction uncertainty.

Vector Targets and Regions

The same conditional logic applies to joint regions. For the coefficient vector, the squared standardized distance from $\hat{\beta}$ to $\beta$ has an $F$ distribution:

Result: Under the model and rank conditions already stated, $$ \frac{(\hat{\beta}-\beta)'G(\hat{\beta}-\beta)}{p\,s^2} \sim F_{p,\nu}. $$ Consequently, the region defined by this quantity being no larger than the $1-\alpha$ quantile of $F_{p,\nu}$ contains $\beta$ with conditional probability $1-\alpha$.

For justification, $G^{1/2}(\hat{\beta}-\beta)/\sigma$ has a $p$-dimensional standard normal distribution, so its squared length has a $\chi^2_p$ distribution. This quantity is independent of $\nu s^2/\sigma^2$, by the projection independence established above. Dividing the first chi-square variable by $p$ and the second by $\nu$ gives the stated $F_{p,\nu}$ statistic.

For $q$ future responses collected in a vector $Y_0$, let $X_0$ be the $q\times p$ matrix of their fixed design rows. Assume their errors are jointly $N(0,\sigma^2 I_q)$, independent of the training errors. Then $Y_0-X_0\hat{\beta}$ has covariance $\sigma^2A$, where

A=I_q+X_0G^{-1}X_0'.

Because $A$ is positive definite and this prediction error is independent of $s^2$, its standardized squared length gives

\frac{(Y_0-X_0\hat{\beta})'A^{-1}(Y_0-X_0\hat{\beta})}{q\,s^2} \sim F_{q,\nu}.

Thus the corresponding ellipsoid has exact conditional coverage $1-\alpha$. The degrees of freedom in the numerator are the dimension $q$ of the future response vector; the denominator degrees of freedom remain the residual degrees of freedom $\nu$.

Worked Example: Checking a Two-Response Prediction Region

Return to the eight-observation design, with $G^{-1}=\operatorname{diag}(1/8,1/28)$, $\nu=6$, and $s^2=4$. Predict two future responses, at predictor values $0$ and $1$, so

X_0= \begin{pmatrix} 1&0\\ 1&1 \end{pmatrix}, \qquad A=I_2+X_0G^{-1}X_0' = \begin{pmatrix} 9/8&1/8\\ 1/8&65/56 \end{pmatrix}.

Suppose the fitted coefficient vector is $(5,0.40)'$, and the two future responses observed are $6.1$ and $7.0$. The fitted means are $5.0$ and $5.4$, so $d=Y_0-X_0\hat{\beta}=(1.1,1.6)'$. Direct inversion gives

A^{-1}=\frac{1}{289} \begin{pmatrix} 260&-28\\ -28&252 \end{pmatrix}, \qquad d'A^{-1}d =\frac{260(1.21)-56(1.76)+252(2.56)}{289} \approx2.980.

The region statistic is $2.980/(2\cdot4)\approx0.373$. Since $F_{2,6;0.95}\approx5.143$, the observed vector lies inside the 95% prediction ellipsoid. This calculation checks membership for these data; the exact 95% coverage is a property of the procedure under repeated sampling with the stated fixed design and independent normal future errors.

When the Conditional Guarantee Applies

If the design matrix is random but independent of the errors and the conditional model assumptions hold, the fixed-design coverage result holds for almost every realized full-rank $X$. Averaging over the distribution of $X$ then also gives unconditional coverage $1-\alpha$. In symbols, for a coverage event $C$,

P(C)=E[P(C\mid X)]=E[1-\alpha]=1-\alpha.

Conditional calibration is especially informative because it does not rely on averaging good and poor coverage across different designs. It says that each design satisfying the assumptions receives the stated coverage on its own.

The normality, common variance, independence, and rank assumptions are substantive. If the errors are heteroskedastic or dependent, or if the future errors are not independent of the training data, the displayed pivots need not have their claimed distributions. Likewise, if $X$ is selected using the outcomes or errors, conditioning on the selected $X$ may alter the conditional error law. Full rank alone does not repair that problem.

Common Pitfall

Wrong reasoning: “The formula uses the observed design matrix, so it must have exact conditional coverage for any way that matrix was obtained.” This confuses using a realized design in a formula with having the assumed sampling distribution conditional on that design.

Correction: Exactness follows from the conditional normal linear model and the independence of the estimator’s projection and the residual projection. If design selection, model selection, or dependence between design and errors changes those conditional distributions, the $t$ and $F$ arguments above do not automatically apply. Check the conditional assumptions before interpreting a nominal coverage level as exact.

Key takeaway: Under a full-rank fixed design, independent normal errors with common variance, and independent future errors when predicting, the usual $t$- and $F$-based procedures have exact finite-sample coverage conditional on the design. The guarantee comes from normal projection theory and independence—not from the formulas alone.

Check Your Understanding

Use the model assumptions and conditional-coverage arguments in this tutorial to answer the following.

Why must the design matrix have full column rank for the stated inverse-matrix formulas?
Which two independent quantities form the $t_\nu$ pivot for a scalar contrast?
Why does the variance factor for predicting one future response include a $1$ in addition to $x_0'G^{-1}x_0$?
For a vector prediction region with $q$ future outcomes, what determines the numerator degrees of freedom of its $F$ statistic?
Why does conditional coverage for every realized design imply the same unconditional coverage when the conditional model assumptions hold?