Joint Confidence Ellipsoids for Mean Responses on a Design Grid

Why a Grid Requires Joint Inference

At a collection of design points, the fitted mean responses are connected: they are all determined by the same regression coefficient vector. A set of separate confidence intervals for those means does not, by itself, give a confidence region for the whole vector. The joint region must account for the covariance among estimated means as well as their individual variances.

The construction follows a direct principle: write the grid means as a linear transformation of the coefficients, then apply that same transformation to the coefficient covariance matrix. In the normal linear model, this yields an exact confidence ellipsoid. The dimension of that ellipsoid is determined by the rank of the grid matrix, not necessarily by the number of grid points.

Model and Grid Notation

Consider the fixed-design normal linear model \(Y=X\beta+\varepsilon\), where \(X\) is an \(n\times p\) matrix of known regressors with full column rank, \(\beta\) is a \(p\)-vector, and the errors are independent \(N(0,\sigma^2)\). Assume \(n>p\), so the residual degrees of freedom \(\nu=n-p\) are positive. Write \(\hat\beta=(X'X)^{-1}X'Y\), \(s^2=\operatorname{SSE}/\nu\), and \(C=(X'X)^{-1}\).

Suppose the target is the vector of mean responses at \(m\) specified design points. Let \(A\) be the \(m\times p\) matrix whose rows are the regression vectors at those points. The target and its estimator are \(\mu_g=A\beta\) and \(\hat\mu_g=A\hat\beta\). By linear transformation, the covariance matrix of the estimator is \(\sigma^2 V\), where \(V=ACA'\). This matrix can be singular: for example, a grid can contain more points than there are independent directions of variation in its mean vector.

Definition: Let \(r=\operatorname{rank}(A)\). The effective dimension of the grid mean vector is \(r\). Since \(C\) is positive definite, \(\operatorname{rank}(V)=\operatorname{rank}(ACA')=r\). If \(r>0\), the grid confidence ellipsoid is a region in the \(r\)-dimensional space \(\operatorname{col}(A)\), even when it is represented as a subset of \(\mathbb R^m\).

For a singular matrix \(V\), \(V^+\) denotes its Moore–Penrose inverse. The expression \((\theta-\hat\mu_g)'V^+(\theta-\hat\mu_g)\) measures squared distance in the estimable directions. Restricting \(\theta\) to \(\operatorname{col}(A)\) is important: the grid mean vector cannot take values outside that space.

The Exact Joint Confidence Region

For \(0<\alpha<1\), let \(F_{r,\nu;1-\alpha}\) denote the \(1-\alpha\) quantile of the central \(F\) distribution with numerator and denominator degrees of freedom \(r\) and \(\nu\). When \(r\geq1\), the exact joint confidence region is:

Theorem: Under the fixed-design normal linear model above, with \(n>p\) and \(r=\operatorname{rank}(A)\geq1\), an exact \(1-\alpha\) confidence region for \(\mu_g=A\beta\) is

\mathcal C_{1-\alpha} = \left\{ \theta\in\operatorname{col}(A): (\theta-\hat\mu_g)'V^+(\theta-\hat\mu_g) \leq r s^2 F_{r,\nu;1-\alpha} \right\}, \qquad V=A(X'X)^{-1}A'.

Here “exact” means \(P(\mu_g\in\mathcal C_{1-\alpha})=1-\alpha\) under the stated model. When \(r=m\), \(V\) is nonsingular and \(V^+=V^{-1}\), so the region has the usual full-dimensional ellipsoid form. When \(r<m\), it is an ellipsoid within the lower-dimensional subspace \(\operatorname{col}(A)\).

Why the F Cutoff Is Exact

The proof identifies the numerator and denominator of an \(F\) statistic. The coefficient estimator is normal with mean \(\beta\) and covariance \(\sigma^2 C\), so \(\hat\mu_g-\mu_g\) is normal with mean zero and covariance \(\sigma^2 V\). Diagonalize \(V\) on its range: it has \(r\) positive eigenvalues and is zero in the remaining directions. After standardizing those \(r\) coordinates, the squared distance has a chi-square distribution with \(r\) degrees of freedom.

For the denominator, the least-squares residuals are the orthogonal projection of \(Y\) onto the complement of the column space of \(X\). That complement has dimension \(\nu=n-p\). Under normal errors, orthogonal projections onto perpendicular subspaces are independent, giving independence of the coefficient-estimation error and the residual sum of squares. Thus \(\operatorname{SSE}/\sigma^2\) has a chi-square distribution with \(\nu\) degrees of freedom and is independent of the numerator.

Standardize the grid-mean error.
Set \(Z=(\hat\mu_g-\mu_g)/\sigma\). It is normal with mean zero and covariance \(V\). In an orthonormal eigenbasis for the range of \(V\), its \(r\) nonzero standardized coordinates are independent \(N(0,1)\) variables. Therefore \(Z'V^+Z\sim\chi^2_r\).

Standardize the residual variance.
The residual projection has rank \(\nu\), so \(\operatorname{SSE}/\sigma^2\sim\chi^2_\nu\). Its projection is orthogonal to the fitted-coefficient projection; joint normality therefore makes the residual sum of squares independent of \(\hat\beta\), and hence of \(Z'V^+Z\).

Form the ratio.
Since \(s^2=\operatorname{SSE}/\nu\), the ratio of the independent chi-square quantities is \(\frac{Z'V^+Z}{r(s^2/\sigma^2)}\sim F_{r,\nu}\). The factor \(\sigma^2\) cancels when the statistic is written using the unstandardized estimation error.

Invert the probability statement.
The probability that this ratio is at most \(F_{r,\nu;1-\alpha}\) is \(1-\alpha\). Replacing the unknown \(\mu_g\) in that inequality by a candidate \(\theta\) gives exactly the displayed region, so the true \(\mu_g\) belongs to it with probability \(1-\alpha\). \(\square\)

The eigenbasis argument also explains why the degrees of freedom are \(r\), not automatically \(m\): only \(r\) independent directions of grid-mean error have positive variance. The denominator degrees of freedom remain \(\nu\), because they come from the residuals in the original regression fit.

Worked Regions

Worked Example: A Two-Point Grid in Simple Linear Regression

Suppose the fitted model has an intercept and slope, with design rows \((1,x)\) at \(x=0,1,2,3\), and observed responses \(1,2,2,4\). Construct the exact 95% confidence region for the mean responses at \(x=0\) and \(x=2\).

Coefficient estimate and residual variance. Here \(X'X=\left(\begin{smallmatrix}4&6\\6&14\end{smallmatrix}\right)\), so \(C=(X'X)^{-1}=\left(\begin{smallmatrix}0.7&-0.3\\-0.3&0.2\end{smallmatrix}\right)\). Also \(X'Y=(9,18)'\), giving \(\hat\beta=CX'Y=(0.9,0.9)'\). The fitted values at \(x=0,1,2,3\) are \(0.9,1.8,2.7,3.6\). The residuals are \(0.1,0.2,-0.7,0.4\), and \(\operatorname{SSE}=0.01+0.04+0.49+0.16=0.70\). Thus \(\nu=4-2=2\) and \(s^2=0.70/2=0.35\).

Transform to the grid. The grid matrix and estimated mean vector are \(A=\left(\begin{smallmatrix}1&0\\1&2\end{smallmatrix}\right)\) and \(\hat\mu_g=A\hat\beta=(0.9,2.7)'\). Direct multiplication gives \(V=ACA'=\left(\begin{smallmatrix}0.7&0.1\\0.1&0.3\end{smallmatrix}\right)\). Its determinant is \(0.7(0.3)-0.1^2=0.20>0\), so \(r=2\) and \(V^{-1}=\left(\begin{smallmatrix}1.5&-0.5\\-0.5&3.5\end{smallmatrix}\right)\).

Write the region. The 95th percentile of \(F_{2,2}\) is \(19\): its cumulative distribution function is \(x/(x+1)\), and \(x/(x+1)=0.95\) gives \(x=19\). Therefore the cutoff is \(r s^2 F_{2,2;0.95}=2(0.35)(19)=13.3\). The region is \[ (\theta-\left(0.9,2.7\right)')' \begin{pmatrix}1.5&-0.5\\-0.5&3.5\end{pmatrix} (\theta-\left(0.9,2.7\right)') \leq13.3. \] The two coordinates are the mean responses at \(x=0\) and \(x=2\), and this single region has 95% joint coverage for that pair.

Worked Example: Three Grid Points but Only One Effective Dimension

Consider an intercept-only model with four observations \(2,3,1,2\). Request the vector of means at three design points. Since the model has no varying regressor, every grid row is \((1)\), so \(A=(1,1,1)'\).

The fitted intercept is \(\hat\beta=(2+3+1+2)/4=2\). The residual sum of squares is \(0^2+1^2+(-1)^2+0^2=2\); hence \(\nu=4-1=3\) and \(s^2=2/3\). The grid mean estimate is \(\hat\mu_g=(2,2,2)'\). Since \(C=(X'X)^{-1}=1/4\), \(V=ACA'=(1/4)\left(\begin{smallmatrix}1&1&1\\1&1&1\\1&1&1\end{smallmatrix}\right)\). This matrix has rank \(r=1\), not 3. Its Moore–Penrose inverse is \(V^+=(4/9)J\), where \(J\) is the \(3\times3\) matrix of ones: \(V\) has eigenvalue \(3/4\) in the direction \((1,1,1)'\) and zero in the two perpendicular directions.

Every possible grid mean has the form \(\theta=(t,t,t)'\). Its squared distance from the estimate is \((\theta-\hat\mu_g)'V^+(\theta-\hat\mu_g)=4(t-2)^2\). The exact 95% region is therefore \(4(t-2)^2\leq(2/3)F_{1,3;0.95}\), or equivalently \(|t-2|\leq\sqrt{F_{1,3;0.95}/6}\), with the three-dimensional representation constrained to the line \(\{(t,t,t)':t\in\mathbb R\}\). Treating these three identical means as three independent directions would give the wrong rank and the wrong cutoff.

Worked Example: A Grid Mean That Is Known Exactly

In a no-intercept model \(E[Y\mid x]=\beta x\), consider a grid consisting of two copies of the design point \(x=0\). Then \(A=\left(\begin{smallmatrix}0\\0\end{smallmatrix}\right)\), so \(r=0\) and \(\mu_g=A\beta=(0,0)'\) for every possible value of \(\beta\).

The grid mean vector is known exactly, regardless of the data or the estimated error variance. Its confidence region can be taken to be the singleton \(\{(0,0)'\}\), which has coverage one. The \(F\)-based formula for positive \(r\) does not apply: \(F_{0,\nu}\) is not the required distribution, and dividing by \(r\) is undefined.

Interpretation and Common Pitfall

This confidence region concerns the vector of expected responses at the specified design points. It does not describe a vector of future observed outcomes. Future outcomes also include random error, so their uncertainty is different; prediction regions are the relevant standard tool for that target. Nor is this region simply a collection of pointwise intervals: its cutoff comes from a joint distribution in \(r\) dimensions and its geometry incorporates covariance across the grid.

Common mistake: using \(m\), the number of grid points, as the numerator degrees of freedom without checking the rank of \(A\). That reasoning assumes the \(m\) means can vary independently. The correction is to calculate \(r=\operatorname{rank}(A)\), use the pseudoinverse when \(r<m\), and restrict the region to \(\operatorname{col}(A)\). If \(r=0\), handle the known-zero mean vector separately as a singleton rather than substituting zero into the positive-rank formula.

Key takeaway: For a nonzero-rank grid, transform the coefficient covariance to \(V=A(X'X)^{-1}A'\) and use the exact cutoff \(r s^2F_{r,\nu;1-\alpha}\). The region’s effective dimension is the rank of the grid matrix; a zero-rank grid has a known mean vector and needs no estimated ellipsoid.

Check Your Understanding

Use the model assumptions stated above when answering.

For a grid matrix \(A\) with four rows and rank two, what are the numerator degrees of freedom in the joint confidence region, and why?
Why must a region for a rank-deficient grid be restricted to \(\operatorname{col}(A)\)?
What does the Moore–Penrose inverse contribute when \(V\) is singular?
In the intercept-only example, why are three grid means represented by a one-dimensional region?
What confidence set is appropriate when \(A\) has rank zero, and why is the positive-rank F formula inapplicable?