Sensitivity of Prediction Intervals to Influential Training Cases

Why Examine Sensitivity?

A prediction interval depends on the regression model fitted to the training data. A single training observation can sometimes have a noticeable effect on the fitted regression coefficients, the estimate of residual variation, or the uncertainty at a particular predictor setting. Consequently, the interval for a new outcome can change when that observation is included or omitted.

That change is useful information, but it is not a deletion rule. An observation may be influential because it has unusual predictor values, an unusual outcome relative to the fitted model, or both. If it is a valid observation from the population the model is meant to describe, its influence may be substantively important rather than a nuisance to remove. The aim here is to compare fits transparently and investigate why they differ.

What Changes When a Case Is Omitted?

For a regression model with \(p\) estimated coefficients, a usual two-sided prediction interval for a new outcome at predictor vector \(\mathbf{x}_0\) has the form

Here \(\hat{y}_0\) is the fitted value at the prediction target, \(s\) is the residual standard error, \(df=n-p\), and \(h_0\) measures the target's leverage in the fitted design. In multiple regression, \(h_0=\mathbf{x}_0^{\mathsf T}(X^{\mathsf T}X)^{-1}\mathbf{x}_0\), using a design matrix \(X\) that includes the intercept when the model has one. The term \(1\) accounts for the variation of a new individual outcome; the leverage term accounts for uncertainty in the fitted mean. This tutorial compares the usual interval from each refitted model, assuming its regression conditions are adequate.

After omitting one case, the number of observations falls by one, and the coefficient estimates generally change. The fitted value \(\hat{y}_0\) may move. The residual standard error \(s\), target leverage \(h_0\), degrees of freedom, and hence the \(t\) critical value may also change. Thus there is no general rule that omission must make an interval narrower or wider.

Why the Width Can Move Either Way

The half-width of the interval is \(M=t_{1-\alpha/2,df}s\sqrt{1+h_0}\). Each factor can move differently after refitting. For example, omitting a case can reduce \(s\), but it can also increase \(h_0\) at the target or slightly increase the critical value by reducing \(df\). Even when the interval narrows, its center can shift; a narrower interval does not necessarily mean that the prediction has become more accurate or that the omitted case was erroneous.

A useful way to compare widths is the ratio of the half-widths:

This follows by dividing the expression for \(M\) after omission by the same expression before omission. It makes clear that a change in width is a combined result, not a direct measure of just one observation's effect on residual variation.

An Influential Case Can Widen the Interval When Included

Influence can also work in the other direction. A valid case may increase the fitted model's residual variation, or shift the prediction target relative to the fitted relationship. Omitting it may make the interval narrower, but that does not show that the narrower interval is preferable: it may instead describe a different population or conceal real variability.

Track the Center and Width Separately

An interval has two distinct features: its center and its width. A case can move the center while barely changing the width, or change the width while leaving the center nearly fixed. Reporting only one feature can therefore hide an important aspect of sensitivity.

Investigate Before Deciding Whether to Omit

A sensitivity comparison is most informative when it follows an explicit investigation. First verify the observation against its source and check units, coding, and data entry. Then ask whether the observation belongs to the population and predictor range the model is intended to address. Examine whether the assumed mean relationship and error behavior remain reasonable with the case included. If a value is a documented error, correction or exclusion may be justified and should be explained. If it is valid, do not remove it merely because it changes the prediction interval.

A valid observation may reveal that the fitted relationship is sensitive in an important region, that the data provide limited support for a prediction, or that a single model does not summarize all cases well. The appropriate response may be to report both fits as a sensitivity analysis, gather more data, or reconsider whether the model is suitable for the stated prediction. Any alternative analysis must still be justified by the question and assumptions.

Common Mistake

A common mistake is to say, “The case is influential, so it should be deleted; the smaller interval after deletion is better.” This reasoning confuses a change in the fit with evidence that the observation is invalid. Influence describes how much a case affects a fitted result; it does not establish that the data are wrong, that the case is outside the population of interest, or that assumptions improve when it is removed.

The correction is to investigate and document the case, then interpret the sensitivity comparison in context. If the observation is valid and relevant, the full-data result generally answers the stated question more directly. If it is an error or does not belong to the target population, an omission may be defensible, but the reason should be stated and the resulting interval interpreted for the population represented by the refitted model.