Leverage and the Width of a Prediction Interval

Why Leverage Matters for a New Outcome

A multiple-regression prediction interval for a new outcome accounts for two kinds of uncertainty. The fitted regression mean at the specified predictor values is estimated from a sample, so that mean is uncertain. In addition, a future individual outcome varies around its mean because of residual variation. Prediction leverage measures the first contribution for a particular predictor profile: it tells us how uncertain the fitted mean is at that profile, relative to the residual variance.

This distinction matters because an interval for a future outcome is not just an interval for the regression mean. Even if the mean were known exactly, a future outcome could differ from it. Conversely, uncertainty in estimating the mean adds to that future-outcome variation. The leverage term expresses the second contribution in a way that depends on the specified predictor values and the observed design.

Model and Definition

Suppose the observed data follow a multiple linear regression model with \(n\) observations and \(p\) regression coefficients, including the intercept. Let \(X\) be the \(n\)-by-\(p\) design matrix, with one row of predictor values for each observation. For a new case, let \(\mathbf{x}_0\) be the \(p\)-by-1 vector of predictor values, including a 1 for the intercept. The model for the new outcome is

Here, \(\boldsymbol{\beta}\) is the vector of regression coefficients. For the usual t-based prediction interval, assume the regression mean is correctly specified and that the errors are independent, have mean zero and a common variance \(\sigma^2\), and are normally distributed. Treat the predictor values, including \(\mathbf{x}_0\), as fixed. Also assume \(X\) has full column rank, so the least-squares coefficient estimates can be calculated. The future error \(\varepsilon_{\text{new}}\) is independent of the errors used to fit the model.

Write \(\hat{\boldsymbol{\beta}}\) for the least-squares coefficient estimates. The fitted mean at the new profile is \(\hat{Y}_0=\mathbf{x}_0^\mathsf{T}\hat{\boldsymbol{\beta}}\). The prediction leverage at this profile is the scalar

Leverage is a property of the predictor profile and the fitted design, not of the future outcome’s residual. It has no measurement units. If the profile is one of the observed cases, this quadratic form gives the corresponding diagonal element of the usual hat matrix. For a new case, it is more precise to call it prediction leverage.

How Leverage Enters the Prediction-Error Variance

The prediction error is the future outcome minus the fitted mean, \(Y_{\text{new}}-\hat{Y}_0\). Substituting the model for the future outcome separates this error into two pieces: the future residual \(\varepsilon_{\text{new}}\), and the error in estimating the mean. Since the new residual is independent of the fitted coefficients, the two pieces have covariance zero.

The coefficient-estimation term follows from the least-squares covariance matrix, \(\operatorname{Var}(\hat{\boldsymbol{\beta}})=\sigma^2(X^\mathsf{T}X)^{-1}\). The expression shows why leverage adds to the future residual variation rather than replacing it: the prediction-error variance contains \(\sigma^2\) for the new residual and \(\sigma^2h_0\) for uncertainty in the fitted mean.

In practice, \(\sigma\) is unknown and is estimated by the residual standard error \(s\). With \(df=n-p\), the usual t-based prediction interval is

Here \(t^*_{df}\) is the positive t critical value for the chosen confidence level and residual degrees of freedom. The standard error for predicting a new outcome is \(s\sqrt{1+h_0}\). By comparison, the standard error for the fitted mean alone is \(s\sqrt{h_0}\). Thus the interval for a new outcome includes both sources of uncertainty.

Comparing Widths at Two Profiles

For a fixed fitted model and confidence level, the interval width is \(2t^*_{df}s\sqrt{1+h_0}\). Therefore, if the residual standard error and degrees of freedom are held fixed, a profile with larger prediction leverage has a larger prediction standard error and a wider prediction interval. This comparison concerns leverage’s contribution to width; it does not imply that the future residual variation itself has changed.

Reading Leverage From a Model Calculation

When the fitted model output does not provide prediction leverage directly, it can be calculated from \(\mathbf{x}_0^\mathsf{T}(X^\mathsf{T}X)^{-1}\mathbf{x}_0\). The intercept must be included in \(\mathbf{x}_0\), and the predictor values must use the same coding and units as the design matrix. Omitting a column or using different coding changes the calculation.

Common Mistakes and a Practical Check

A common mistake is to use \(s\sqrt{h_0}\) as the standard error for a new outcome. That is the standard error for estimating the fitted mean, not for predicting one future outcome. It omits the new case’s residual variation. The correction is to use \(s\sqrt{1+h_0}\) for a prediction interval. Another mistake is to treat leverage as an estimate of residual noise: leverage describes how much the estimated mean’s uncertainty contributes at a profile, while \(s\) estimates the model’s residual scale.

Before interpreting an interval, check that its leverage was calculated for the specified predictor profile, that the intercept and predictor coding match the fitted model, and that the interval is for a new individual outcome rather than a conditional mean. Also report the outcome’s units. Leverage alone does not determine whether the model is appropriate for prediction; the stated regression assumptions still matter.