Confidence Intervals for Gaussian Mean Responses

Introduction

A fitted Gaussian regression model gives a conditional mean response at every covariate vector. That fitted mean is an estimate: another sample could produce a different estimate, even if the covariate vector is held fixed. A confidence interval for a mean response quantifies this estimation uncertainty at one specified covariate vector.

The interval is not for a future individual response. A future response varies both because the mean is estimated and because individual outcomes vary around that mean. Here we focus on the first source of uncertainty: inference for $x_0^{\mathsf T}\beta$, the model’s conditional mean at a fixed $x_0$.

Model and Target

Assume $n$ independent responses follow the identity-link Gaussian model

Y_i\mid x_i \sim N(x_i^{\mathsf T}\beta,\phi/w_i),\qquad i=1,\ldots,n,

where $x_i$ is a fixed $p$-dimensional covariate vector, $\beta$ is a $p$-dimensional coefficient vector, $\phi>0$ is the dispersion, and the known precision weights satisfy $w_i>0$. Let $X$ be the $n\times p$ matrix with rows $x_i^{\mathsf T}$, and let $W=\operatorname{diag}(w_1,\ldots,w_n)$. Assume $X$ has full column rank $p$. The weighted least squares estimator is

\hat\beta=(X^{\mathsf T}WX)^{-1}X^{\mathsf T}WY.

For a fixed covariate vector $x_0$, the target conditional mean and its estimate are

\mu_0=x_0^{\mathsf T}\beta,\qquad \hat\mu_0=x_0^{\mathsf T}\hat\beta.

Definition: A pointwise $100(1-\alpha)\%$ confidence interval for $x_0^{\mathsf T}\beta$ is a random interval constructed from the data whose repeated-sampling coverage probability is $1-\alpha$ for that specified, fixed $x_0$, under the stated model assumptions.

Define the leverage-like quantity for this target as

c_0=x_0^{\mathsf T}(X^{\mathsf T}WX)^{-1}x_0.

Since $X^{\mathsf T}WX$ is positive definite and $x_0\ne 0$, $c_0>0$. If $x_0=0$, the target is identically zero and its interval is degenerate.

Known Dispersion and Estimated Dispersion

The coefficient estimator has covariance matrix $\phi(X^{\mathsf T}WX)^{-1}$, so the variance of the estimated mean is $\phi c_0$. When $\phi$ is known, a standard normal quantile gives an exact interval. When it is unknown, estimate it from the weighted residual sum of squares

Q=(Y-X\hat\beta)^{\mathsf T}W(Y-X\hat\beta) =\sum_{i=1}^n w_i(Y_i-x_i^{\mathsf T}\hat\beta)^2,\qquad \hat\phi=\frac{Q}{n-p}.

The degrees-of-freedom corrected estimate is used for the exact interval: under the Gaussian model, $Q/\phi$ has a chi-square distribution with $n-p$ degrees of freedom and is independent of $\hat\beta$. The maximum-likelihood estimate $Q/n$ is not the dispersion estimate in this exact Student t formula.

Theorem: Suppose the observations are independent Gaussian responses with the model and full-rank design specified above, and $n>p$. For fixed $x_0$, if $\phi$ is known, an exact $100(1-\alpha)\%$ confidence interval for $x_0^{\mathsf T}\beta$ is $$ \hat\mu_0\ \pm\ z_{1-\alpha/2}\sqrt{\phi c_0}, $$ where $z_{1-\alpha/2}$ is the $1-\alpha/2$ quantile of the standard normal distribution. If $\phi$ is unknown, an exact interval is $$ \hat\mu_0\ \pm\ t_{n-p,1-\alpha/2}\sqrt{\hat\phi c_0}, $$ where $t_{n-p,1-\alpha/2}$ is the corresponding Student t quantile.

Proof. Write the model in vector form as $Y=X\beta+\varepsilon$, where $\varepsilon\sim N(0,\phi W^{-1})$. Then

\hat\beta-\beta=(X^{\mathsf T}WX)^{-1}X^{\mathsf T}W\varepsilon.

This is a linear transformation of a Gaussian vector, so it is Gaussian with mean zero. Its covariance is

(X^{\mathsf T}WX)^{-1}X^{\mathsf T}W (\phi W^{-1})W X(X^{\mathsf T}WX)^{-1} =\phi(X^{\mathsf T}WX)^{-1}.

Consequently, $\hat\mu_0-\mu_0$ is normal with mean zero and variance $\phi c_0$. If $\phi$ is known, standardizing gives a standard normal random variable, which proves the stated normal interval has coverage $1-\alpha$.

For unknown $\phi$, set $Z=W^{1/2}X$ and $\epsilon=W^{1/2}\varepsilon/\sqrt{\phi}$. Then $\epsilon\sim N(0,I_n)$, and the weighted residual sum of squares satisfies

\frac{Q}{\phi} =\epsilon^{\mathsf T}(I-P_Z)\epsilon,\qquad P_Z=Z(Z^{\mathsf T}Z)^{-1}Z^{\mathsf T}.

The matrix $P_Z$ is the orthogonal projection onto the $p$-dimensional column space of $Z$. Thus $I-P_Z$ is an orthogonal projection of rank $n-p$. Applying an orthogonal change of coordinates that diagonalizes this projection shows that $Q/\phi$ is a sum of squares of $n-p$ independent standard normal variables; hence it has a chi-square distribution with $n-p$ degrees of freedom. The estimator $\hat\beta$ depends only on the projection of $\epsilon$ onto the column space of $Z$, while $Q$ depends only on its orthogonal complement. These two projections are independent: their coordinates in that orthogonal basis are disjoint groups of independent normal variables. Therefore $\hat\mu_0$ and $Q$ are independent. It follows that

\frac{\hat\mu_0-\mu_0}{\sqrt{\hat\phi c_0}} = \frac{(\hat\mu_0-\mu_0)/\sqrt{\phi c_0}} {\sqrt{(Q/\phi)/(n-p)}} \sim t_{n-p}.

Inverting this exact t probability statement gives the asserted interval. $\square$

Worked Examples

Worked Example: An Unweighted Linear Regression

Suppose $n=5$, $p=2$, and the observations are $(x_i,y_i)=(0,1),(1,2),(2,2),(3,4),(4,6)$. Fit a straight-line mean with an intercept and unit weights, and find a 95% interval for the mean response at $x_0=2$.

The sample means are $\bar x=2$ and $\bar y=3$. The slope is $S_{xy}/S_{xx}=12/10=1.2$, so the intercept is $3-1.2(2)=0.6$. Thus the fitted mean at $x_0=2$ is $0.6+1.2(2)=3$.

The fitted values at $x=0,1,2,3,4$ are $0.6,1.8,3,4.2,5.4$. The residuals are $0.4,0.2,-1,-0.2,0.6$, and therefore $Q=0.16+0.04+1+0.04+0.36=1.6$. With $n-p=3$, $\hat\phi=1.6/3=0.533333\ldots$.

For simple linear regression with unit weights, $c_0=1/n+(x_0-\bar x)^2/S_{xx}$. Here $x_0=\bar x=2$, so $c_0=1/5=0.2$. The standard error is $\sqrt{(1.6/3)(0.2)}=\sqrt{0.106666\ldots}\approx0.326599$. Using $t_{3,0.975}\approx3.182446$, the margin is approximately $1.03938$. The interval is $3\pm1.03938$, or approximately $[1.961,4.039]$.

Worked Example: A Weighted Intercept-Only Model

Consider four independent Gaussian observations with a common mean, weights $(1,2,1,2)$, and unknown dispersion. Take the observed responses to be $11/3+a,\ 11/3-a,\ 11/3+a,\ 11/3$, where $a=\sqrt{11/6}$. Find the 95% confidence interval for the mean.

The total weight is $1+2+1+2=6$. The weighted deviations from $11/3$ sum to $a-2a+a+0=0$, so the weighted least squares estimate of the common mean is $\hat\mu_0=11/3$. The weighted residual sum of squares is $Q=a^2+2a^2+a^2=4a^2=4(11/6)=22/3$. With $n-p=4-1=3$, $\hat\phi=(22/3)/3=22/9$.

For an intercept-only design, $X^{\mathsf T}WX=6$ and $x_0=1$, giving $c_0=1/6$. The standard error is $\sqrt{(22/9)(1/6)}=\sqrt{11/27}\approx0.638285$. Using $t_{3,0.975}\approx3.182446$, the margin is approximately $2.031307$. Thus the interval is $11/3\pm2.031307$, or approximately $[1.635,5.698]$ to three decimal places.

Worked Example: Known Dispersion in an Intercept-Only Model

Suppose four independent responses have a common Gaussian mean, unit weights, and known dispersion $\phi=4$. The observed responses are $2,4,6,8$. Find a 95% confidence interval for their common mean.

The estimate is the sample mean, $\hat\mu_0=(2+4+6+8)/4=5$. The target is the intercept, so $c_0=1/4$, and the standard error is $\sqrt{4(1/4)}=1$. The normal critical value is $z_{0.975}\approx1.96$. The interval is $5\pm1.96$, or $[3.04,6.96]$. There is no t critical value here because the dispersion is specified as known.

What This Interval Includes—and Excludes

The standard error $\sqrt{\hat\phi c_0}$ (or $\sqrt{\phi c_0}$ when dispersion is known) measures uncertainty in the estimated conditional mean. It depends on the design, the precision weights, and the specified $x_0$. In general, mean estimates are less precise at covariate vectors that are poorly supported by the design, because the corresponding $c_0$ can be larger.

The interval does not add the residual variance of a new response. Its target is the fixed quantity $x_0^{\mathsf T}\beta$, not a random future observation. This is why its standard error uses $c_0$ but no additional $1$ term. A prediction interval for a new Gaussian response has a different target and must account for response variation around the mean.

Coverage is pointwise: the stated probability applies at the particular fixed $x_0$. It does not, by itself, guarantee that an entire collection of intervals across many covariate vectors will cover all corresponding means simultaneously. The exact coverage statements also rely on the Gaussian model, independent observations, the stated variance structure, and a full-rank design.

Common Mistake

Adding residual variation to a confidence interval for the mean. It is incorrect to treat the target as a new observed response and inflate the standard error for this interval. The mean-response interval measures uncertainty in $x_0^{\mathsf T}\hat\beta$, so its variance estimate is $\hat\phi c_0$. Residual response variation belongs to the distinct task of predicting a new response.
Using a normal critical value after estimating dispersion. If $\phi$ is unknown and estimated by $Q/(n-p)$, the exact Gaussian-model interval uses $t_{n-p,1-\alpha/2}$, not $z_{1-\alpha/2}$.
Using $n$ rather than $n-p$ in the exact t interval. The residual sum of squares has $n-p$ degrees of freedom after fitting $p$ coefficients. The exact interval uses $\hat\phi=Q/(n-p)$.
Calling pointwise intervals simultaneous. Each interval has its stated coverage for its specified covariate vector. A collection of pointwise intervals does not automatically have that same coverage for all mean responses at once.

Key takeaway: For a fixed $x_0$, estimate the mean by $x_0^{\mathsf T}\hat\beta$, calculate $c_0=x_0^{\mathsf T}(X^{\mathsf T}WX)^{-1}x_0$, and use the normal interval for known dispersion or the exact $t_{n-p}$ interval when dispersion is estimated. These intervals concern the mean, not a future response.

Check Your Understanding

Try these questions before moving on.

In a Gaussian model with known $\phi$, what is the standard error of the estimated mean at a fixed $x_0$?
Why does an exact interval with estimated dispersion use $n-p$ degrees of freedom?
For a weighted intercept-only model with total weight $8$, what is $c_0$ for the common mean?
In the first worked example, why is the target at $x_0=2$ estimated more precisely than it might be at a covariate value far from the observed $x$-values?
Explain why a confidence interval for a mean response does not include an additional term for residual response variation.