Confidence Intervals for Linear Combinations

Why Confidence Intervals for Linear Combinations Matter

A regression question may concern more than one coefficient at a time. For example, a model might contain separate slopes for two predictors, while the quantity of interest is the difference between those slopes. Or a model with an interaction may imply that a slope at a particular setting is the sum of two coefficients. In each case, the target is a linear combination of regression coefficients, and its uncertainty depends on the uncertainty and covariance of the estimates involved.

A confidence interval for a linear combination uses the same basic structure as an interval for one coefficient: estimate plus or minus a critical value times a standard error. The key is to calculate the estimate and standard error for the whole combination, rather than combining separate intervals for its coefficients.

Definition: Let the regression coefficient vector be \(\boldsymbol{\beta}\), and let \(\mathbf{c}\) be a fixed vector of known weights. A linear contrast is \(L=\mathbf{c}^{\mathsf T}\boldsymbol{\beta}\). Its estimate is \(\hat{L}=\mathbf{c}^{\mathsf T}\hat{\boldsymbol{\beta}}\). The weights may be positive, negative, or zero; a contrast need not be a difference whose weights sum to zero.

Model and Confidence-Interval Formula

Use the multiple linear regression model \(Y=X\boldsymbol{\beta}+\boldsymbol{\varepsilon}\), where \(Y\) is the vector of responses, \(X\) is a fixed design matrix of full column rank, and the errors are independent with mean zero and common variance \(\sigma^2\). For the exact t-based interval, also assume normally distributed errors. If the model has \(p\) fitted coefficients, including the intercept, and \(n\) observations, the residual degrees of freedom are \(n-p\).

Write \(\widehat{\operatorname{Cov}}(\hat{\boldsymbol{\beta}})\) for the estimated covariance matrix of the coefficient estimates. Under the model above, it is \(s^2(X^{\mathsf T}X)^{-1}\), where \(s^2\) is the residual mean square. The estimate and standard error for a contrast are:

\hat{L}=\mathbf{c}^{\mathsf T}\hat{\boldsymbol{\beta}}, \qquad \widehat{\operatorname{SE}}(\hat{L}) = \sqrt{\mathbf{c}^{\mathsf T} \widehat{\operatorname{Cov}}(\hat{\boldsymbol{\beta}}) \mathbf{c}}.

For a two-sided confidence level \(1-\alpha\), the interval is:

\hat{L}\ \pm\ t_{1-\alpha/2,\,n-p} \widehat{\operatorname{SE}}(\hat{L}).

Here \(t_{1-\alpha/2,\,n-p}\) is the corresponding quantile of the t distribution with \(n-p\) degrees of freedom. The interval is exact under the stated normal-error model. Without normal errors, the formula may still be used as a large-sample approximation in appropriate settings, but it is not generally an exact finite-sample interval.

Why the Contrast Standard Error Has This Form

The variance calculation follows from the rule for the variance of a weighted sum. If \(\hat{\boldsymbol{\beta}}\) has covariance matrix \(V\), then the scalar \(\mathbf{c}^{\mathsf T}\hat{\boldsymbol{\beta}}\) has variance \(\mathbf{c}^{\mathsf T}V\mathbf{c}\). Expanding this expression shows why covariance terms cannot usually be omitted:

\operatorname{Var}(\mathbf{c}^{\mathsf T}\hat{\boldsymbol{\beta}}) = \sum_j c_j^2\operatorname{Var}(\hat{\beta}_j) + 2\sum_{j<k}c_jc_k\operatorname{Cov}(\hat{\beta}_j,\hat{\beta}_k).

Under the normal linear regression model, the coefficient estimates are jointly normal. The contrast estimate is therefore normal with mean \(L\) and variance \(\sigma^2\mathbf{c}^{\mathsf T}(X^{\mathsf T}X)^{-1}\mathbf{c}\). Replacing \(\sigma^2\) by \(s^2\) introduces the residual-variance estimate. The regression result that the scaled residual variance has a chi-square distribution with \(n-p\) degrees of freedom, independently of the coefficient estimates, gives the t statistic:

\frac{\hat{L}-L} {\widehat{\operatorname{SE}}(\hat{L})} \sim t_{n-p}.

Rearranging the central probability statement for this t distribution gives the interval formula above. Its center is \(\hat{L}\), and its half-width is the critical value multiplied by the contrast standard error.

A Reliable Calculation Process

Write the target as a contrast.
List the coefficient weights in \(\mathbf{c}\), matching their order to the coefficient vector.

Calculate the contrast estimate.
Multiply each estimated coefficient by its weight and add the results.

Calculate its variance and standard error.
Use the full estimated covariance matrix, including off-diagonal terms, and take the square root of the variance.

Choose the critical value and form the endpoints.
Use the stated confidence level and residual degrees of freedom \(n-p\). Add and subtract the margin of error from the contrast estimate.

Worked Example: A Confidence Interval for a Difference in Slopes

A hypothetical regression for annual claim-processing time has two estimated slopes, \(\hat{\beta}_1=2.6\) and \(\hat{\beta}_2=0.7\), measured in the same units. The estimated standard errors are 0.15 and 0.10, and the estimated covariance of the two coefficient estimates is \(-0.005\). There are \(n=30\) observations and \(p=6\) fitted coefficients. Construct a 95% confidence interval for \(L=\beta_1-\beta_2\), using \(t_{0.975,24}=2.064\).

Method and contrast estimate. The contrast weights are \((1,-1)\), so \(\hat{L}=1(2.6)-1(0.7)=1.9\). The residual degrees of freedom are \(n-p=30-6=24\), as specified by the critical value.

Contrast variance. The variance of the difference includes minus twice the covariance:

\widehat{\operatorname{Var}}(\hat{L}) = 0.15^2+0.10^2-2(-0.005) =0.0225+0.0100+0.0100 =0.0425.

Thus, \(\widehat{\operatorname{SE}}(\hat{L})=\sqrt{0.0425}=0.2061552813\) (using extra digits for the endpoint calculation). The margin of error is \(2.064(0.2061552813)=0.4255045\). The unrounded endpoints are \(1.9-0.4255045=1.4744955\) and \(1.9+0.4255045=2.3255045\). Rounded to three decimals, the 95% interval is \([1.474,\ 2.326]\).

Check and common mistake avoided. The endpoints are centered at 1.9, the contrast estimate. The negative covariance increases the variance of a difference because the covariance term is \(-2\operatorname{Cov}(\hat{\beta}_1,\hat{\beta}_2)\). Dropping that term or treating it as \(+2(-0.005)\) would give the wrong standard error.

Using a Covariance Matrix for Several Coefficients

When a contrast contains three or more coefficients, matrix notation helps keep the weights and covariance terms organized. The diagonal entries of the covariance matrix are the coefficient-estimate variances, while the off-diagonal entries are covariances. The following calculation shows how all of them contribute.

Worked Example: Combining Three Regression Coefficients

A hypothetical model for a benefit-cost measure has coefficient estimates \((0.5,0.2,0.1)^{\mathsf T}\). For the contrast \(L=\beta_1-\beta_2+\beta_3\), the estimated covariance matrix is:

\widehat{\operatorname{Cov}}(\hat{\boldsymbol{\beta}}) = \begin{pmatrix} 0.0100&0.0020&0.0010\\ 0.0020&0.0064&-0.0010\\ 0.0010&-0.0010&0.0144 \end{pmatrix}.

There are \(n=28\) observations and \(p=7\) fitted coefficients, and \(t_{0.975,21}=2.080\). Find a 95% confidence interval.

Method and estimate. The contrast vector is \(\mathbf{c}=(1,-1,1)^{\mathsf T}\). Therefore, \(\hat{L}=0.5-0.2+0.1=0.4\). The residual degrees of freedom are \(28-7=21\).

Contrast variance. The diagonal contribution is \(0.0100+0.0064+0.0144=0.0308\). The three distinct off-diagonal contributions, before doubling, are \(1(-1)(0.0020)=-0.0020\), \(1(1)(0.0010)=0.0010\), and \((-1)(1)(-0.0010)=0.0010\). They sum to zero, so:

\widehat{\operatorname{Var}}(\hat{L}) =0.0308+2(-0.0020+0.0010+0.0010) =0.0308.

The standard error is \(\sqrt{0.0308}=0.1754993\), and the margin of error is \(2.080(0.1754993)=0.3650385\). The endpoints are \(0.4-0.3650385=0.0349615\) and \(0.4+0.3650385=0.7650385\). Rounded to three decimals, the interval is \([0.035,\ 0.765]\).

Check and common mistake avoided. The interval is centered at 0.4, not at the sum of the absolute coefficient estimates. Applying the negative weight to \(\hat{\beta}_2\) is essential both in the estimate and in the covariance terms.

Covariance Signs and Other Exam Traps

For a difference, the covariance term has a minus sign in the expanded variance formula. For a sum, it has a plus sign. But the covariance itself can be negative, so the safest approach is always to substitute the contrast weights into \(2c_jc_k\operatorname{Cov}(\hat{\beta}_j,\hat{\beta}_k)\) rather than relying on a verbal rule.

Worked Example: Checking the Covariance Sign in a Difference

A hypothetical regression gives \(\hat{\beta}_1=1.0\), \(\hat{\beta}_2=0.6\), standard errors 0.15 and 0.10, and covariance \(-0.003\). Construct a 90% interval for \(L=\beta_1-\beta_2\), given \(n=25\), \(p=5\), and \(t_{0.95,20}=1.725\).

Method and estimate. The contrast weights are \((1,-1)\), so \(\hat{L}=1.0-0.6=0.4\). The degrees of freedom are \(25-5=20\).

Variance and interval. Substituting the weights gives:

\widehat{\operatorname{Var}}(\hat{L}) =0.15^2+0.10^2-2(-0.003) =0.0225+0.0100+0.0060 =0.0385.

The standard error is \(\sqrt{0.0385}=0.1962142\). The margin of error is \(1.725(0.1962142)=0.3384695\). The endpoints are \(0.4-0.3384695=0.0615305\) and \(0.4+0.3384695=0.7384695\). Rounded to three decimals, the 90% interval is \([0.062,\ 0.738]\).

Common mistake and correction. A tempting error is to calculate \(0.15^2+0.10^2+2(-0.003)=0.0265\). That expression uses the covariance sign for a sum, not the weights \((1,-1)\). The correct cross term is \(2(1)(-1)(-0.003)=0.006\), giving variance \(0.0385\). The final interval is centered at 0.4, which also checks the estimate.

Another frequent error is adding coefficient standard errors directly. Standard errors are not generally additive; first calculate the variance of the contrast using the covariance matrix, then take its square root. Also distinguish \(p\), the number of fitted coefficients, from the number of predictors when the intercept is included in the model. Using the wrong residual degrees of freedom changes the t critical value.

Interpreting the Interval

A 95% confidence procedure has a repeated-sampling interpretation: if the model and sampling assumptions hold and the same method is applied to many comparable samples, about 95% of the resulting intervals would contain the fixed contrast \(L\). For the data at hand, the computed interval gives a range of values compatible with the estimate and its sampling uncertainty under the model.

The target is the population linear combination \(L=\mathbf{c}^{\mathsf T}\boldsymbol{\beta}\), not a new observation and not a separate interval for each coefficient. An interval for \(\beta_1-\beta_2\), for example, directly quantifies uncertainty about the slope difference. It is not obtained by subtracting the endpoints of two separate coefficient intervals, because those intervals do not account correctly for the joint uncertainty and covariance.

Key takeaway: For a t-based confidence interval for a regression contrast, calculate \(\hat{L}=\mathbf{c}^{\mathsf T}\hat{\boldsymbol{\beta}}\), calculate its standard error from the full estimated covariance matrix, and use \(t_{1-\alpha/2,n-p}\). Keep the contrast weights, covariance signs, interval center, and residual degrees of freedom consistent.

Check Your Understanding

For each problem, calculate the contrast estimate first, then its covariance-aware standard error and interval where requested. Round interval endpoints to three decimals.

For \(L=2\beta_1+\beta_2\), suppose \(\hat{\beta}_1=0.4\), \(\hat{\beta}_2=0.3\), their standard errors are 0.10 and 0.20, and their covariance is \(-0.004\). Use \(n=22\), \(p=6\), and \(t_{0.975,16}=2.120\) to find a 95% interval. (Answer: \([0.564,\ 1.636]\))
For \(L=\beta_1-2\beta_2\), suppose the estimates are 0.8 and 0.2, the standard errors are 0.12 and 0.08, and the covariance is 0.002. Use \(n=32\), \(p=7\), and \(t_{0.975,25}=2.060\) to find a 95% interval. (Answer: \([0.031,\ 0.769]\))
For \(L=\beta_1+\beta_3\), suppose the estimates are 0.25 and 0.15, their standard errors are 0.05 and 0.09, and their covariance is \(-0.001\). Use \(n=18\), \(p=4\), and \(t_{0.95,14}=1.761\) to find a 90% interval. (Answer: \([0.237,\ 0.563]\))
For \(L=\beta_1+\beta_2-\beta_3\), the estimate vector is \((0.6,0.1,0.2)^{\mathsf T}\), and the estimated covariance matrix is \(\begin{pmatrix}0.0100&0.0010&-0.0020\\0.0010&0.0064&0.0010\\-0.0020&0.0010&0.0144\end{pmatrix}\). Use \(n=24\), \(p=6\), and \(t_{0.975,18}=2.101\) to find a 95% interval. (Answer: \([0.108,\ 0.892]\))
A 95% interval for a regression contrast is \([1.2,2.0]\). State its center and margin of error, and explain what repeated-sampling statement the 95% refers to. (Answer: Center \(1.6\), margin of error \(0.4\); under the model and sampling assumptions, about 95% of intervals produced by the same procedure in repeated comparable samples would contain the fixed contrast.)

What You'll Learn