Why Put a Range Around a Regression Coefficient?
A fitted regression coefficient is an estimate based on a particular sample. If another sample were collected under the same model, its estimated coefficient would generally differ. A confidence interval quantifies this sampling uncertainty by giving a range of values compatible with the observed estimate and its estimated standard error.
This tutorial concerns an interval for one population regression coefficient at a time. It is not an interval for the average response at a selected predictor value, a prediction interval for a future response, or a simultaneous range intended to cover several coefficients together.
Model, Standard Error, and Interval Formula
Consider a multiple linear regression with \(n\) observations and \(k\) predictors, including an intercept:
Write \(\mathbf{X}\) for the \(n\) by \(k+1\) design matrix, whose first column is the intercept column. Assume that the predictor values are fixed, or condition on the observed design matrix. For an exact t-based interval, the errors must be independent and normally distributed with mean zero and common variance \(\sigma^2\). The design matrix must have full column rank, and \(n>k+1\), so that the coefficients are identifiable and there are positive residual degrees of freedom.
Let \(\widehat{\beta}_j\) denote the least-squares estimate of \(\beta_j\). Let \(C_{jj}\) be the diagonal entry corresponding to that coefficient in \((\mathbf{X}^{\mathsf T}\mathbf{X})^{-1}\). The residual sum of squares is \(\operatorname{SSE}=\sum_{i=1}^n (Y_i-\widehat{Y}_i)^2\). The residual degrees of freedom are \(n-k-1\), and the estimated error variance is \(s^2=\operatorname{SSE}/(n-k-1)\). Thus the estimated standard error of \(\widehat{\beta}_j\) is \(s\sqrt{C_{jj}}\).
Why the t Critical Value Appears
The interval uses a t critical value because the error variance is unknown and is estimated from the residuals. The justification combines the sampling distribution of the coefficient estimate with the distribution of the residual variance estimate.
Under the model assumptions, \(\widehat{\beta}_j\) is normally distributed with mean \(\beta_j\) and variance \(\sigma^2 C_{jj}\). Therefore, \((\widehat{\beta}_j-\beta_j)/(\sigma\sqrt{C_{jj}})\) has a standard normal distribution.
Normal errors and least squares imply that \(\operatorname{SSE}/\sigma^2\) has a chi-square distribution with \(n-k-1\) degrees of freedom. This quantity is independent of the standardized coefficient estimate.
Dividing the standard normal quantity by the square root of the independent chi-square quantity divided by its degrees of freedom gives \((\widehat{\beta}_j-\beta_j)/(s\sqrt{C_{jj}})\sim t_{n-k-1}\).
The probability that a t variable with \(n-k-1\) degrees of freedom lies between \(-t_{1-\alpha/2,n-k-1}\) and \(t_{1-\alpha/2,n-k-1}\) is \(1-\alpha\). Substituting the coefficient statistic and rearranging for \(\beta_j\) gives the stated interval.
This argument also shows why the assumptions matter. Without normal errors, the exact finite-sample t distribution need not hold. If the errors have unequal variances or are dependent, the displayed standard error and its t-based justification may not apply. If \(n=k+1\), the residual degrees of freedom are zero, so \(s^2\) cannot be estimated by this formula and the stated interval is undefined.
Constructing and Interpreting an Interval
A reliable calculation keeps the variance estimate, coefficient-specific standard error, and critical value as separate steps. The coefficient index \(j\) matters: each coefficient has its own \(C_{jj}\), and therefore usually its own standard error. The degrees of freedom, however, are \(n-k-1\) for every coefficient in the same fitted model.
Worked Example: A Slope in a Hypothetical Claims Model
In a hypothetical regression of annual claim cost on one predictor, a fitted model uses \(n=12\) policies and has \(k=1\) predictor. The estimated slope for the predictor is \(1.800\), \(\operatorname{SSE}=20\), and the corresponding entry of \((\mathbf{X}^{\mathsf T}\mathbf{X})^{-1}\) is \(C_{jj}=0.050\). Construct a 95% interval, using \(t_{0.975,10}=2.228\), and interpret it. Assume the regression conditions for an exact interval hold.
Method and degrees of freedom. This is an individual-coefficient t interval. The residual degrees of freedom are \(n-k-1=12-1-1=10\), matching the supplied critical value.
Estimate the error variance. We have \(s^2=20/10=2\), so \(s=\sqrt{2}\).
Find the slope standard error. The estimated standard error is \(s\sqrt{C_{jj}}=\sqrt{2}\sqrt{0.050}=\sqrt{0.100}\approx0.3162278\).
Find the margin and endpoints. The margin is \(2.228(0.3162278)\approx0.7045\). The interval is \(1.800\pm0.7045\), giving endpoints \(1.0955\) and \(2.5045\). Rounded only at the final step to three decimals, the interval is \((1.095,\ 2.505)\).
Interpretation and check. Under the model assumptions, this procedure gives intervals containing the population slope in 95% of repeated samples. Here the plausible values for the slope extend from about 1.095 to 2.505, in the response units per unit of the predictor. The endpoints are equally far from 1.800, as required by the symmetric formula. This calculation avoids using the number of observations, rather than the residual degrees of freedom, to select the t critical value.
Worked Example: A Coefficient Adjusted for Other Predictors
A hypothetical multiple regression models a monthly benefit amount using three predictors. From \(n=20\) observations, the estimated coefficient of one predictor is \(-0.400\), \(\operatorname{SSE}=48\), and its diagonal inverse-matrix entry is \(C_{jj}=0.090\). Construct a 90% interval using \(t_{0.95,16}=1.746\). Assume the exact-interval conditions hold.
Method and degrees of freedom. The interval concerns one coefficient in the multiple regression, so use its own \(C_{jj}\). The residual degrees of freedom are \(20-3-1=16\).
Estimate variance and standard error. The residual variance estimate is \(s^2=48/16=3\). Thus \(s=\sqrt{3}\), and the coefficient standard error is \(\sqrt{3(0.090)}=\sqrt{0.270}\approx0.5196152\).
Calculate the margin. The margin is \(1.746(0.5196152)\approx0.9072\). Therefore, the endpoints are \(-0.400-0.9072=-1.3072\) and \(-0.400+0.9072=0.5072\).
Answer and interpretation. Rounded to three decimals, the 90% interval is \((-1.307,\ 0.507)\). It describes plausible values for this population coefficient, holding the other predictors in the model fixed in the coefficient interpretation. It does not say that 90% of individual benefit amounts fall within this range. The calculation avoids treating the coefficient as if it came from a regression with only one predictor.
Worked Example: An Interval for the Intercept
A hypothetical regression of a laboratory outcome on two predictors has \(n=15\), \(\operatorname{SSE}=26\), an estimated intercept of \(5.200\), and \(C_{00}=0.040\) for the intercept’s diagonal entry. Construct a 99% interval using \(t_{0.995,12}=3.055\). Assume the exact-interval conditions hold.
Method and degrees of freedom. There are \(k=2\) predictors, so \(n-k-1=15-2-1=12\) residual degrees of freedom. The subscript \(00\) identifies the intercept’s entry, rather than a slope’s entry.
Estimate the variance and standard error. We calculate \(s^2=26/12=13/6\approx2.1666667\). The standard error is \(\sqrt{(13/6)(0.040)}=\sqrt{0.0866667}\approx0.2943920\).
Calculate the interval. The margin is \(3.055(0.2943920)\approx0.8994\). The lower endpoint is \(5.200-0.8994=4.3006\), and the upper endpoint is \(5.200+0.8994=6.0994\). Rounded to three decimals, the interval is \((4.301,\ 6.099)\).
Interpretation and check. The interval concerns the population intercept: the mean response when all predictors equal zero, if that setting is meaningful in the application. Its midpoint is \(5.200\), and its half-width is approximately \(0.899\). The high confidence level uses a larger critical value than a 95% interval would use with the same degrees of freedom.
Common Mistakes and Exam Traps
Small changes in the inputs can change an interval substantially. Keep the coefficient estimate, variance estimate, coefficient-specific matrix entry, and degrees of freedom visible until the calculation is complete.
- Using the wrong residual degrees of freedom. With an intercept and \(k\) predictors, use \(n-k-1\), not \(n-1\), \(n-k\), or \(n\), for both \(s^2\) and the t critical value.
- Using \(\operatorname{SSE}\) directly as the variance estimate. Estimate \(\sigma^2\) by \(\operatorname{SSE}/(n-k-1)\). The standard error uses \(s=\sqrt{s^2}\), not \(s^2\).
- Using a generic standard error for every coefficient. The standard error for coefficient \(j\) is \(s\sqrt{C_{jj}}\), using that coefficient’s own diagonal entry.
- Using a one-sided critical value for a two-sided interval. A two-sided \(100(1-\alpha)\%\) interval uses \(t_{1-\alpha/2,n-k-1}\), not \(t_{1-\alpha,n-k-1}\).
- Interpreting the confidence level as a probability about a fixed parameter. In the repeated-sampling interpretation, the parameter is fixed and the interval is random. After observing the data, one reports the calculated endpoints as plausible parameter values under the model.
- Forgetting the rank and positive-degrees-of-freedom conditions. Full column rank makes the coefficients identifiable; \(n>k+1\) is also needed for a positive residual degrees of freedom and this residual-based t interval.
Worked Example: Checking the Degrees of Freedom and Rounding
A hypothetical model has \(n=14\), \(k=3\), \(\operatorname{SSE}=30\), an estimate \(\widehat{\beta}_j=1.100\), and \(C_{jj}=0.160\). Use \(t_{0.975,10}=2.228\) to construct a 95% interval. Assume the design matrix has full column rank and the exact-interval conditions hold.
Method and degrees of freedom. The residual degrees of freedom are \(14-3-1=10\), which are positive and match the supplied t quantile.
Estimate the variance and standard error. We obtain \(s^2=30/10=3\). Then \(s\sqrt{C_{jj}}=\sqrt{3(0.160)}=\sqrt{0.480}\approx0.6928203\).
Calculate the margin and endpoints. The margin is \(2.228(0.6928203)\approx1.543603\). The lower endpoint is \(1.100-1.543603=-0.443603\); the upper endpoint is \(1.100+1.543603=2.643603\). Rounded to three decimals, the interval is \((-0.444,\ 2.644)\).
Interpretation and check. The interval is centered at 1.100, and each endpoint is approximately 1.544 from the estimate. Rounding each endpoint to three decimals only after computing it gives the stated result. Rounding the standard error or margin too early can change the final endpoint by one unit in the last reported decimal place.
What an Individual Interval Does—and Does Not—Say
The interval describes uncertainty about one selected coefficient under the specified model. Its interpretation is conditional on the included predictors, their coding, and the regression assumptions. For example, a slope in a multiple regression represents the modeled change in mean response for a one-unit change in that predictor while the other predictors are held fixed.
A \(100(1-\alpha)\%\) interval is designed to have coverage \(1-\alpha\) for that coefficient when the procedure is repeated under the assumptions. Constructing separate \(95\%\) intervals for several coefficients does not, by itself, make the probability that all of them are covered equal to \(95\%\). Each interval has its stated individual coverage; simultaneous coverage is a different objective.
A useful final check is to confirm that the interval is centered on \(\widehat{\beta}_j\), that the margin is positive, that the t degrees of freedom equal \(n-k-1\), and that the reported units match the coefficient. Inference based on an interval is only as dependable as the model assumptions and the design that support it.
Check Your Understanding
For each problem, use the stated t quantile and round interval endpoints to three decimals.
- A hypothetical simple regression has \(n=11\), \(k=1\), \(\operatorname{SSE}=18\), \(\widehat{\beta}_1=2.400\), and \(C_{11}=0.080\). Construct a 95% interval using \(t_{0.975,9}=2.262\). (Answer: \((1.495,\ 3.305)\))
- A hypothetical model with two predictors has \(n=18\), \(\operatorname{SSE}=45\), an estimated coefficient of \(-0.750\), and its \(C_{jj}=0.040\). Construct a 90% interval using \(t_{0.95,15}=1.753\). (Answer: \((-1.357,\ -0.143)\))
- A hypothetical model has \(n=16\), \(k=3\), \(\operatorname{SSE}=24\), an estimated intercept of \(4.000\), and \(C_{00}=0.125\). Construct a 95% interval using \(t_{0.975,12}=2.179\). (Answer: \((2.911,\ 5.090)\))
- A hypothetical regression has \(n=14\), \(k=3\), \(\operatorname{SSE}=30\), \(\widehat{\beta}_j=1.100\), and \(C_{jj}=0.160\). Using \(t_{0.975,10}=2.228\), give the 95% interval and state whether zero is inside it. (Answer: \((-0.444,\ 2.644)\); zero is inside.)
- A regression includes an intercept and three predictors, with \(n=4\) observations. Can the residual-based t interval in this tutorial be constructed? Give the residual degrees of freedom and explain. (Answer: No; \(4-3-1=0\), so the residual variance estimate and t interval are undefined.)