Leverage and Mean-Response Region Width

Why Some Mean-Response Intervals Are Wider

A fitted regression surface does not estimate its mean response equally precisely at every design point. Even when the same residual standard deviation and confidence level are used, some pointwise intervals are wider than others. The geometric quantity responsible is the quadratic form \(x_0^{\mathsf T}(X^{\mathsf T}X)^{-1}x_0\): it is the variance factor for estimating the mean response at \(x_0\).

At a row of the design matrix, this quantity is the usual diagonal leverage of that observation. A larger value means that the fitted mean at that point has greater sampling variance, and therefore a wider pointwise confidence interval when the other ingredients are held fixed. The comparison is about the uncertainty in a mean response, not about the variability of an individual future outcome.

Model, Leverage, and Interval Width

Assume the fixed-design normal linear model \(Y=X\beta+\varepsilon\), where \(X\) is an \(n\times p\) matrix of fixed, known design values with rank \(p\), \(n>p\), and \(\varepsilon\sim N_n(0,\sigma^2I_n)\), with \(\sigma^2>0\). The least-squares estimator is \(\hat\beta=(X^{\mathsf T}X)^{-1}X^{\mathsf T}Y\). For a specified vector \(x_0\in\mathbb R^p\), the mean response is \(\mu_0=x_0^{\mathsf T}\beta\), estimated by \(\hat\mu_0=x_0^{\mathsf T}\hat\beta\). Let \(s^2=\operatorname{SSE}/(n-p)\), where \(\operatorname{SSE}=(Y-X\hat\beta)^{\mathsf T}(Y-X\hat\beta)\).

The inverse \( (X^{\mathsf T}X)^{-1}\) is positive definite because \(X\) has full column rank. Consequently \(q_0>0\) for every nonzero \(x_0\), and \(q_0=0\) exactly when \(x_0=0\). The interval width follows from the variance of \(\hat\mu_0\) and the exact \(t\) calibration.

Why the Variance Factor Controls Width

First, write \(\hat\beta-\beta=(X^{\mathsf T}X)^{-1}X^{\mathsf T}\varepsilon\). Since the error covariance matrix is \(\sigma^2I_n\), this gives \(\operatorname{Cov}(\hat\beta)=\sigma^2(X^{\mathsf T}X)^{-1}\). Multiplying by \(x_0^{\mathsf T}\) and \(x_0\) gives the variance of the fitted mean.

For completeness, the exact calibration uses the residual variance estimate as well as this variance factor. Define \(A=(X^{\mathsf T}X)^{-1}X^{\mathsf T}\), so \(\hat\beta-\beta=A\varepsilon\), and let \(M=I_n-H\). The residual vector is \(MY\), because \(MX=0\), and \(\operatorname{SSE}=\varepsilon^{\mathsf T}M\varepsilon\). The matrix \(M\) is symmetric and idempotent, with rank \(n-p\). Thus the normal projection result gives \(\operatorname{SSE}/\sigma^2\sim\chi^2_{n-p}\).

The estimator \(A\varepsilon\) and the residual vector \(M\varepsilon\) are jointly normal. They are independent because their cross-covariance is zero: \(A(\sigma^2I_n)M=\sigma^2AM=0\), as \(AM=(X^{\mathsf T}X)^{-1}X^{\mathsf T}(I_n-H)=0\). When \(q_0>0\), it follows that \((\hat\mu_0-\mu_0)/(\sigma\sqrt{q_0})\sim N(0,1)\), independently of \((n-p)s^2/\sigma^2\sim\chi^2_{n-p}\). Dividing the first quantity by the square root of the second divided by its degrees of freedom yields the \(t_{n-p}\) pivot.

The central \(t\) quantile therefore puts probability \(1-\alpha\) between its two signed critical values. Solving those two inequalities for \(\mu_0\) gives the stated interval. Its endpoints differ by \(2t_{n-p,\,1-\alpha/2}s\sqrt{q_0}\), proving the width formula. If \(q_0=0\), positive definiteness implies \(x_0=0\), so the target is identically zero and the pivot division is not defined; this is why that case must be treated separately. \(\square\)

Leverage Across the Design

At an observed design point \(x_i^{\mathsf T}\), the interval width is \(2t_{n-p,\,1-\alpha/2}s\sqrt{h_{ii}}\). Thus, for two observed points \(i\) and \(j\), within the same fitted model and at the same confidence level, their widths have ratio \(\sqrt{h_{ii}/h_{jj}}\), provided both leverages are positive. The square root matters: doubling leverage multiplies width by \(\sqrt{2}\), not by 2.

The diagonal leverages also have a useful global constraint. Since \(H\) is an idempotent projection of rank \(p\), its trace is \(p\). The trace is the sum of its diagonal elements, so the average observed-point leverage is \(p/n\).

To verify the result, use the trace identity \(\operatorname{tr}(BC)=\operatorname{tr}(CB)\) for conformable matrices:

Dividing by \(n\) proves the average claim. \(\square\)

Reading Width Comparisons Carefully

The formula isolates the geometric contribution to precision: \(q_0\) enters the half-width through \(\sqrt{q_0}\). Comparisons across points in the same design hold the degrees of freedom and critical value fixed, as well as the common residual estimate \(s\). If comparing separately fitted models, those other quantities may differ too, so the quadratic form alone need not determine which interval is wider.

A common mistake is to say that an interval at a point with twice the leverage must be twice as wide. The half-width is proportional to the square root of leverage, so the multiplier is \(\sqrt{2}\). Another mistake is to apply the exact \(1-\alpha\) coverage statement to \(x_0=0\): there the target is known to equal zero and the degenerate interval covers with probability 1. Finally, the exact interval here concerns the mean response. An interval for a future individual outcome has a different variance calculation and is not obtained merely by relabeling this mean-response interval.