Regions With Known and Estimated Error Variance

Why the Error-Variance Choice Changes a Region

A confidence or prediction region depends on the uncertainty of its estimator or predictor. In a Gaussian linear model, that uncertainty is proportional to the error variance \(\sigma^2\). If \(\sigma^2\) is known, Gaussian quantiles give exact coverage. If it is unknown, the residual variance \(s^2\) must be estimated, adding uncertainty to the calibration. The exact correction is a Student \(t\) quantile for one scalar target and an \(F\) quantile for a joint region.

The distinction is not simply a choice between two critical values. The variance estimate is random, and its precision depends on the residual degrees of freedom. The \(t\) and \(F\) distributions account for that extra randomness. In a regression model with \(n\) observations and \(p\) linearly independent design columns, the residual degrees of freedom are \(\nu=n-p\).

Model and Region Definitions

Condition on a fixed design matrix \(X\) of size \(n\times p\) and rank \(p\). Assume the response vector satisfies \(Y=X\beta+\varepsilon\), where \(\varepsilon\sim N_n(0,\sigma^2 I_n)\). The least-squares estimator is \(\hat\beta=(X^\mathsf{T}X)^{-1}X^\mathsf{T}Y\), and the residual variance estimator is \(s^2=\mathrm{RSS}/\nu\), where \(\mathrm{RSS}=(Y-X\hat\beta)^\mathsf{T}(Y-X\hat\beta)\) and \(\nu=n-p\).

Consider a \(q\)-dimensional estimable target \(\theta=A\beta\), where \(A\) has \(q\) linearly independent rows. Define \(C=A(X^\mathsf{T}X)^{-1}A^\mathsf{T}\), which is positive definite under these conditions. Then \(\hat\theta=A\hat\beta\) has conditional covariance \(\sigma^2 C\). For a proposed value of \(\theta\), its squared distance from the estimate in this covariance metric is \(Q(\theta)=(\hat\theta-\theta)^\mathsf{T}C^{-1}(\hat\theta-\theta)\).

Definition: For a \(1-\alpha\) confidence region, a known-variance Gaussian calibration uses the cutoff \(\chi^2_{q,1-\alpha}\) for \(Q(\theta)/\sigma^2\). When \(\sigma^2\) is unknown and estimated by \(s^2=\mathrm{RSS}/\nu\), the exact calibration uses the cutoff \(qF_{q,\nu;1-\alpha}\) for \(Q(\theta)/s^2\). Here \(\chi^2_{q,1-\alpha}\) and \(F_{q,\nu;1-\alpha}\) denote the \(1-\alpha\) quantiles of the indicated chi-square and \(F\) distributions.

For \(q=1\), write \(C=c\), a positive scalar. The known-variance region is the interval \(\hat\theta\pm z_{1-\alpha/2}\sigma\sqrt{c}\), where \(z_{1-\alpha/2}\) is the standard normal quantile. With estimated variance, it becomes \(\hat\theta\pm t_{\nu,1-\alpha/2}s\sqrt{c}\). The scalar \(t\) result is the one-dimensional version of the \(F\) result because \(F_{1,\nu}=t_\nu^2\).

Why the Estimated-Variance Cutoff Is Exact

The key facts are specific to the Gaussian linear model. The standardized estimator error has a multivariate normal distribution, the residual sum of squares divided by \(\sigma^2\) has a chi-square distribution with \(\nu\) degrees of freedom, and these two quantities are independent. The independence is essential: it allows the random residual variance in the denominator to produce an exact \(F\) pivot.

Theorem: Under the fixed-design Gaussian model above, for an estimable \(q\)-dimensional target with positive-definite \(C\), \[ \frac{Q(\theta)}{\sigma^2}\sim\chi^2_q,\qquad \frac{Q(\theta)/q}{s^2}\sim F_{q,\nu},\quad \nu=n-p. \] Consequently the two regions \[ Q(\theta)\leq \sigma^2\chi^2_{q,1-\alpha} \quad\text{and}\quad Q(\theta)\leq q s^2 F_{q,\nu;1-\alpha} \] have conditional coverage \(1-\alpha\), respectively when \(\sigma^2\) is known and when it is estimated.

Proof. Since \(\hat\beta\sim N_p(\beta,\sigma^2(X^\mathsf{T}X)^{-1})\), the vector \(C^{-1/2}(\hat\theta-\theta)/\sigma\) has distribution \(N_q(0,I_q)\). Its squared length is therefore chi-square with \(q\) degrees of freedom, proving the first distributional statement.

Let \(P_X=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}\), the projection onto the column space of \(X\). The residual vector is \((I_n-P_X)Y=(I_n-P_X)\varepsilon\). The matrix \(I_n-P_X\) is symmetric and idempotent, with rank \(n-p=\nu\). Orthogonal diagonalization of this projection, together with the spherical normal distribution of \(\varepsilon\), shows that \(\mathrm{RSS}/\sigma^2\) is the sum of squares of \(\nu\) independent standard normal variables, hence has a \(\chi^2_\nu\) distribution. The fitted estimator error is determined by the projection of \(\varepsilon\) onto the column space of \(X\), while the residual is determined by its orthogonal complement. These projections are independent for a multivariate normal vector, so \(Q(\theta)\) and \(s^2\) are independent.

By definition, if \(U\sim\chi^2_q\) and \(V\sim\chi^2_\nu\) are independent, then \((U/q)/(V/\nu)\sim F_{q,\nu}\). Here \(Q(\theta)/\sigma^2\) plays the role of \(U\), and \(\mathrm{RSS}/\sigma^2=\nu s^2/\sigma^2\) plays the role of \(V\). Thus \[ \frac{Q(\theta)/q}{s^2} = \frac{(Q(\theta)/\sigma^2)/q}{(\mathrm{RSS}/\sigma^2)/\nu} \sim F_{q,\nu}. \] Taking the corresponding quantiles gives the stated regions and their coverage. \(\square\)

Worked Regions and Their Cutoffs

Worked Example: A Scalar Mean-Response Interval

Suppose a regression has \(n=12\) observations and \(p=2\) fitted coefficients. At a design point, the estimated mean response is \(8\), and its variance factor is \(c=0.25\). Compare 95% intervals when \(\sigma=2\) is known and when the residual estimate is \(s^2=4\).

The residual degrees of freedom are \(\nu=12-2=10\). With known variance, the standard error is \(\sigma\sqrt{c}=2\sqrt{0.25}=1\). Using \(z_{0.975}=1.960\), the interval is \(8\pm1.960(1)=[6.040,9.960]\).

When the variance is estimated, \(s\sqrt{c}=2\sqrt{0.25}=1\). The 97.5th percentile of \(t_{10}\) is approximately \(2.228\), so the exact interval is \(8\pm2.228(1)=[5.772,10.228]\).

The variance estimate happens to equal the known variance in this numerical comparison, but the estimated-variance interval is wider because the \(t_{10}\) cutoff accounts for the uncertainty in estimating \(\sigma^2\). With a different observed \(s\), its width would also change.

Worked Example: A Joint Two-Contrast Region

Suppose the target is a pair of estimable contrasts with estimate \(\hat\theta=(1,2)^\mathsf{T}\) and covariance factor \(C=\operatorname{diag}(0.25,1)\). Let \(n=12\), \(p=2\), and compare a 95% region using known \(\sigma^2=4\) with one using \(s^2=4\).

The degrees of freedom are \(\nu=10\), and the quadratic form is \[ Q(\theta)=\frac{(\hat\theta_1-\theta_1)^2}{0.25}+(\hat\theta_2-\theta_2)^2. \] For known variance, \(\chi^2_{2,0.95}\approx5.991\). The cutoff is \(4(5.991)=23.964\), so the region is \(Q(\theta)\leq23.964\).

For estimated variance, \(F_{2,10;0.95}\approx4.103\). The cutoff is \(2(4)(4.103)=32.824\), so the region is \(Q(\theta)\leq32.824\). To check the geometry, the semi-axis lengths are the square roots of the cutoff times the corresponding diagonal entries of \(C\). They are approximately \((2.448,4.895)\) for the known-variance region and \((2.864,5.729)\) for the estimated-variance region.

Both regions are centered at \((1,2)^\mathsf{T}\), but their calibration is joint: the cutoff is based on a two-dimensional chi-square or \(F\) distribution, not on two separate 95% scalar intervals.

Worked Example: A Prediction Interval for One Future Response

Suppose a regression has \(n=8\), \(p=2\), and the fitted mean at a new design point is \(12\). Let the leverage at that point be \(h_0=0.25\). For one future response, the prediction error variance is \(\sigma^2(1+h_0)\): the additional \(1\) represents the new observation's error, while \(h_0\) represents estimation uncertainty in the fitted mean. Compare 95% intervals for known \(\sigma^2=4\) and estimated \(s^2=4\).

There are \(\nu=8-2=6\) residual degrees of freedom. The known-variance prediction standard error is \(\sqrt{4(1+0.25)}=\sqrt{5}\approx2.236\). Thus the interval is \(12\pm1.960(2.236)\), or approximately \([7.618,16.382]\).

With estimated variance, the standard error factor is \(s\sqrt{1+h_0}=\sqrt{4}\sqrt{1.25}=\sqrt{5}\). Since \(t_{6,0.975}\approx2.447\), the exact interval is \(12\pm2.447(2.236)\), or approximately \([6.529,17.471]\).

The prediction target changes the variance factor, not the degrees-of-freedom rule: the residual variance still uses \(n-p=6\). The interval is wider than a mean-response interval at the same point because it includes the variability of a future observation.

Applying the Same Calibration to Regions

For a vector of future responses at a fixed design matrix \(X_0\), the prediction error is \(Y_0-X_0\hat\beta\). Under independent future errors, its covariance is \(\sigma^2 C_{\mathrm{pred}}\), where \(C_{\mathrm{pred}}=X_0(X^\mathsf{T}X)^{-1}X_0^\mathsf{T}+I\). When this matrix is positive definite, the same argument gives a chi-square region with known \(\sigma^2\) and an \(F_{q,\nu}\)-calibrated region with estimated variance. The target covariance factor changes, but the denominator degrees of freedom remain \(\nu=n-p\) because the estimate \(s^2\) comes from the original regression residuals.

This distinction helps keep the sources of uncertainty straight. The covariance factor describes the target: a mean response, a contrast, or future observations. The residual degrees of freedom describe how much information is available to estimate the common error variance. Changing the target can change the shape and scale of the region without changing \(\nu\).

Common Mistakes

Using \(n\) rather than \(n-p\) degrees of freedom. The residual sum of squares has \(\nu=n-p\) degrees of freedom because fitting \(p\) independent coefficients uses \(p\) dimensions.
Using a scalar \(t\) cutoff for a joint region. A collection of scalar intervals does not automatically have the desired joint coverage. For a \(q\)-dimensional ellipsoid, use \(qF_{q,\nu;1-\alpha}\) with the stated covariance metric.
Assuming an estimated-variance interval must always be wider than a known-variance interval. For a given data set, the estimated width depends on the observed \(s\). The \(t\) or \(F\) calibration accounts for variance-estimation uncertainty; it does not force every realized interval to be wider than one based on a particular supplied value of \(\sigma\).
Using the same variance factor for a mean and a future observation. A future observation has its own error in addition to uncertainty in the estimated mean. Use the prediction variance factor for prediction targets.

Key takeaway: With known \(\sigma^2\), Gaussian errors yield exact normal or chi-square calibration. With \(\sigma^2\) estimated from regression residuals, use \(t\) for one scalar target or \(F_{q,n-p}\) for a \(q\)-dimensional region. The target determines the covariance factor; the fitted model determines the residual degrees of freedom.

Check Your Understanding

Answer each question by identifying the target dimension, variance choice, and relevant degrees of freedom.

A regression has \(n=20\) observations and \(p=4\) coefficients. What are the residual degrees of freedom for an exact interval based on \(s^2\)?
Why does a \(q\)-dimensional region with estimated variance use an \(F_{q,\nu}\) distribution rather than a scalar \(t_\nu\) distribution?
For a one-dimensional estimable target with variance factor \(c\), write the 95% interval when \(\sigma\) is known and when the variance is estimated.
In a prediction region for multiple future responses, what part of the covariance accounts for the future observations' own errors?
Can an interval based on an observed \(s\) be narrower than one based on a supplied \(\sigma\)? Explain the role of the observed variance estimate and the \(t\) cutoff.