Tutorials › Regression Analysis (Undergraduate) › Deriving the Prediction-Error Variance

Home  /  Tutorials  /  Prediction Intervals for New Outcomes

Prediction Intervals for New Outcomes · Tutorial 719 of 991

Deriving the Prediction-Error Variance

Learn to separate the two sources of prediction error and combine their variances correctly under the regression error assumptions.

Intermediate 10 min read

What You'll Learn

  • Define prediction error as a future response minus its fitted mean.
  • State the independent, constant-variance error assumptions behind the variance result.
  • Derive the sum of future-error variance and fitted-mean estimation variance.
  • Calculate prediction-error variance from the two component variances.
  • Explain why a prediction concerns an individual outcome, not just its conditional mean.
  • Identify how dependence can add a covariance term and change the result.

Why Prediction Error Has Two Sources

A fitted regression mean describes the model’s expected response at a specified predictor value. A future individual response can differ from that mean for two reasons: the fitted mean is estimated from a finite training sample, and the future individual has an error of their own. A prediction must account for both sources of variation.

For example, a model might estimate the mean follow-up measurement for patients with a particular baseline score. Even if that mean were known exactly, an individual patient’s measurement would still vary around it. In practice, the mean is not known exactly either; it is estimated using the training data. The variance of prediction error captures these two contributions.

Definition: Let \(Y_0\) be the response of a new case at a specified predictor value, and let \(\hat{\mu}_0\) be the fitted regression mean at that value. The prediction error is \(Y_0-\hat{\mu}_0\). Its variance describes the variability of this error over repeated training samples and future outcomes, under the stated regression model.

Model and Assumptions

Write the regression model for the training observations as \(Y_i=\mu_i+\varepsilon_i\), where \(\mu_i\) is the model mean for case \(i\). For a future case at the predictor value of interest, write \(Y_0=\mu_0+\varepsilon_0\). The following assumptions are central to the variance derivation:

  • The regression mean model is correctly specified for the predictor values being considered.
  • Conditional on the predictors, each error has mean zero and the same variance \(\sigma^2\).
  • Training errors are independent of one another, and the future error \(\varepsilon_0\) is independent of the training errors.
  • The predictor values are treated as fixed for the variance calculation.

In particular, the future error has \(\operatorname{Var}(\varepsilon_0)=\sigma^2\). The fitted mean \(\hat{\mu}_0\) is calculated from the training data, so its estimation error, \(\hat{\mu}_0-\mu_0\), varies from one training sample to another. Under the assumptions above, the future error is independent of that estimation error.

The result concerns the variance of the prediction error, not the variance of the fitted mean alone. The earlier material on fitted-mean uncertainty concerns \(\operatorname{Var}(\hat{\mu}_0)\); prediction adds the variability of a new individual response.

Deriving the Prediction-Error Variance

Subtract the fitted mean from the future response. Adding and subtracting the true mean \(\mu_0\) separates the future error from the estimation error:

$$ Y_0-\hat{\mu}_0 =(\mu_0+\varepsilon_0)-\hat{\mu}_0 =\varepsilon_0-(\hat{\mu}_0-\mu_0). $$

The variance of a difference is the sum of the two variances minus twice their covariance. Since \(\varepsilon_0\) is independent of the training data, its covariance with the fitted-mean estimation error is zero. Also, multiplying a random quantity by \(-1\) does not change its variance. Therefore:

$$ \begin{aligned} \operatorname{Var}(Y_0-\hat{\mu}_0) &=\operatorname{Var}(\varepsilon_0) +\operatorname{Var}(\hat{\mu}_0-\mu_0) -2\operatorname{Cov}(\varepsilon_0,\hat{\mu}_0-\mu_0)\\ &=\sigma^2+\operatorname{Var}(\hat{\mu}_0). \end{aligned} $$

The final equality uses the fact that \(\mu_0\) is fixed at the specified predictor value, so \(\operatorname{Var}(\hat{\mu}_0-\mu_0)=\operatorname{Var}(\hat{\mu}_0)\). This is the variance decomposition: future-error variance plus fitted-mean estimation variance. Both terms are variances on the response scale, so they can be added directly. Standard deviations, by contrast, are not added directly.

Theorem: Under a correctly specified regression mean model with independent, mean-zero errors of constant variance \(\sigma^2\), and with the future error independent of the training data, \(\operatorname{Var}(Y_0-\hat{\mu}_0)=\sigma^2+\operatorname{Var}(\hat{\mu}_0)\).

Worked Variance Calculations

Worked Example: Combining the Two Variance Components

Suppose a hypothetical regression model predicts a patient’s follow-up measurement in millimetres of mercury (mmHg). At a specified baseline profile, assume the future-error standard deviation is \(4.00\) mmHg and the standard error of the fitted mean is \(1.50\) mmHg. Assume independent, constant-variance errors and independence between the future case and the training data. Find the prediction-error variance and standard deviation.

First convert each standard deviation to a variance. The future-error variance is \(4.00^2=16.0\) \((\text{mmHg})^2\). The fitted-mean estimation variance is \(1.50^2=2.25\) \((\text{mmHg})^2\). Thus:

$$ \operatorname{Var}(Y_0-\hat{\mu}_0) =16.0+2.25 =18.25\;(\text{mmHg})^2. $$

The prediction-error standard deviation is the square root of this variance: \(\sqrt{18.25}=4.272\) mmHg. The calculation shows why the prediction-error standard deviation is larger than the future-error standard deviation alone: it includes uncertainty in the fitted mean as well as individual variation.

Worked Example: Fitted-Mean Uncertainty at a Predictor Value

Consider five hypothetical training cases with baseline scores \(6, 9, 10, 11,\) and \(14\) points. A simple linear regression uses baseline score to predict a follow-up score. For this variance illustration, treat the error variance as known to be \(\sigma^2=25\) score-points squared, and suppose a new case has baseline score \(x_0=13\). The errors are assumed independent with constant variance, and the new case is independent of the training cases.

For simple linear regression, the fitted-mean variance at \(x_0\) is \(\sigma^2\left(\frac{1}{n}+\frac{(x_0-\bar{x})^2}{S_{xx}}\right)\), where \(S_{xx}=\sum_i(x_i-\bar{x})^2\). From the five predictor values, \(n=5\) and \(\bar{x}=(6+9+10+11+14)/5=10\). The sum of squared deviations is:

$$ S_{xx}=(6-10)^2+(9-10)^2+(10-10)^2+(11-10)^2+(14-10)^2 =16+1+0+1+16=34. $$

Therefore, the fitted-mean estimation variance at \(x_0=13\) is:

$$ \operatorname{Var}(\hat{\mu}_0) =25\left(\frac{1}{5}+\frac{(13-10)^2}{34}\right) =25\left(0.2000+\frac{9}{34}\right) =25(0.4647) =11.62\;\text{score-points}^2. $$

The future-error variance is \(25\;\text{score-points}^2\), so:

$$ \operatorname{Var}(Y_0-\hat{\mu}_0) =25+11.62 =36.62\;\text{score-points}^2. $$

The prediction-error standard deviation is \(\sqrt{36.62}=6.051\) score points. The fitted-mean variance depends on the training predictor values and the new predictor value; it is distinct from the future individual’s error variance. This calculation gives a variance, not a prediction interval.

Why Independence Matters

The zero-covariance step in the derivation is not automatic. It follows from the assumption that the future error is independent of the training data. If the future error and the fitted-mean estimation error are dependent, the general variance identity instead gives:

$$ \operatorname{Var}(Y_0-\hat{\mu}_0) =\sigma^2+\operatorname{Var}(\hat{\mu}_0) -2\operatorname{Cov}(\varepsilon_0,\hat{\mu}_0-\mu_0). $$

The covariance term can increase or decrease the prediction-error variance. The simple sum is justified only when that covariance is zero. Independence is a sufficient condition for zero covariance; merely knowing that two quantities have mean zero is not enough.

Worked Example: A Dependence Term Changes the Answer

Suppose, hypothetically, that a future assay error \(\varepsilon_0\) and one training error \(\varepsilon_1\) both have variance \(16\) assay-units squared, but they are correlated because of a shared batch effect. Let the fitted-mean estimation error be \(A=\hat{\mu}_0-\mu_0=0.5\varepsilon_1\), and suppose \(\operatorname{Cov}(\varepsilon_0,\varepsilon_1)=8\) assay-units squared. These conditions deliberately violate independence between the future error and training data.

The fitted-mean estimation variance and covariance are:

$$ \operatorname{Var}(A)=0.5^2(16)=4, \qquad \operatorname{Cov}(\varepsilon_0,A)=0.5(8)=4. $$

Using the variance formula for a difference:

$$ \operatorname{Var}(\varepsilon_0-A) =16+4-2(4)=12\;\text{assay-units}^2. $$

Simply adding the two variances would give \(16+4=20\), which is incorrect for this dependent example. The covariance term reduces the variance to \(12\) assay-units squared. Under the ordinary independent-error assumptions, that covariance is zero and the sum-of-variances result applies.

Interpretation and Common Mistakes

The two terms represent different questions. The future-error variance, \(\sigma^2\), describes how much individual outcomes vary around their true regression means. The fitted-mean estimation variance describes how much the estimated mean at the new predictor value varies across training samples. A prediction for one new case must include both.

  • Adding standard errors rather than variances. If the component standard errors are \(4.00\) and \(1.50\) mmHg, the variance is \(4.00^2+1.50^2\), not \(4.00+1.50\). Take the square root only after adding variances.
  • Using only fitted-mean uncertainty. That omits the new individual’s error and describes uncertainty about a mean, not prediction error for an individual outcome.
  • Using only the residual variation. That ignores uncertainty from estimating the fitted mean with a finite sample.
  • Assuming the sum always holds. The covariance term vanishes under the stated independence condition; dependence requires the full variance identity.

In applications, \(\sigma^2\) is usually unknown and is estimated from the regression data. The theoretical decomposition still clarifies what a prediction procedure must account for: estimated future-response variation and fitted-mean uncertainty. The subsequent calculation of a prediction interval also requires a suitable calibration method; the variance derivation by itself does not supply interval endpoints or coverage.

Key takeaway: With independent, constant-variance regression errors, the prediction error for a new response has variance equal to the future error variance plus the variance of the fitted mean. Keep the two sources distinct, add their variances, and check the independence condition that removes the covariance term.

Check Your Understanding

Answer each question using the variance decomposition and its assumptions.

  1. In your own words, distinguish future-error variance from fitted-mean estimation variance.
  2. A future-error standard deviation is \(3\) units and the fitted-mean standard error is \(2\) units. What is the prediction-error variance, and what is its standard deviation?
  3. Why does the variance derivation require the future error to be independent of the training data?
  4. If the covariance between the future error and fitted-mean estimation error is \(c\), write the variance of their difference.
  5. Why does a prediction for an individual outcome generally have greater variance than uncertainty about the fitted mean alone?

Previous

Next

Back to Tutorials