Why Sensitivity Analysis Is Not Automatic Deletion
An unusual observation can have a large effect on a least-squares fit, but an outlier or influence flag is a reason to investigate, not proof that the observation is wrong. Deleting every flagged case would make the analysis depend on a diagnostic rule rather than on the question and the data’s provenance. It can also hide genuine variation that the regression is meant to describe.
Sensitivity analysis makes the consequences of an analytical choice visible. The primary analysis uses all observations that are valid for the stated target. A sensitivity analysis then compares that fit with one or more defensible alternatives, such as a fit excluding a flagged case for diagnostic comparison or a fit using a reasonable alternative specification. If the estimates or conclusions change, the change is part of the result to report—not a reason to quietly select the preferred-looking fit.
Set Up the Comparison Before Interpreting It
For simple linear regression, the least-squares line fitted to all \(n\) observations is \(\hat{y}=b_0+b_1x\), where \(b_0\) is the estimated intercept and \(b_1\) is the estimated slope. In multiple regression, the fitted response is a linear combination of the predictors and their estimated coefficients. In either setting, a sensitivity comparison needs a clearly identified response, predictors, units, target population or range, primary fit, and alternative fit.
The usual least-squares model assumes that the conditional mean response is represented adequately by the chosen regression form, that errors have mean zero given the predictors, and that observations are independent. Standard model-based inference also relies on assumptions about error variance and, for exact small-sample tests and intervals, an error distribution. These assumptions concern how the model is used; they do not imply that every observation must look typical. A sensitivity fit can reveal fragility, but it does not by itself establish that one model’s assumptions are correct.
Before comparing fits, establish whether the flagged record is a valid observation for the analysis. Check data entry, units, eligibility, timing, and how the response and predictors were measured. If a verified recording error is corrected, document the correction. If the observation is outside the target population or fails a rule defined independently of the outcome, exclusion may be justified for that target. Otherwise, retain it in the primary fit and use a case-excluded fit only as a diagnostic comparison.
A useful comparison looks beyond whether a coefficient crosses a conventional significance threshold. Examine the coefficient’s direction, magnitude, and units; predictions at substantively relevant predictor values; and, when inferential conclusions matter, standard errors and intervals as well. A tiny numerical change can be important in one context and immaterial in another. Decide what change would matter scientifically or practically rather than relying on a universal cutoff.
Worked Examples
Worked Example: A Flagged Endpoint in a Small Data Set
Suppose a hypothetical laboratory exercise records predictor \(x\), in hours, and response \(y\), in concentration units, for five valid samples. The final sample is flagged for review because it lies at the largest observed \(x\)-value. The data are:
| Sample | \(x\) (hours) | \(y\) (concentration units) |
|---|---|---|
| 1 | 0 | 1.48 |
| 2 | 1 | 1.84 |
| 3 | 2 | 2.00 |
| 4 | 3 | 4.16 |
| 5, flagged | 4 | 4.52 |
Assume the records have been checked and all five samples are valid. The primary fit therefore retains all five. For simple linear regression, the slope is \(b_1=S_{xy}/S_{xx}\), where \(S_{xx}=\sum (x_i-\bar{x})^2\) and \(S_{xy}=\sum (x_i-\bar{x})(y_i-\bar{y})\). The means are \(\bar{x}=2\) hours and \(\bar{y}=2.8\) concentration units. The centered \(x\)-values are \(-2,-1,0,1,2\); the centered \(y\)-values are \(-1.32,-0.96,-0.80,1.36,1.72\). Thus:
The full-data slope and intercept are therefore:
The primary fitted line is \(\hat{y}=1.12+0.84x\). Its estimated slope is 0.84 concentration units per hour. At \(x=3\) hours, the predicted response is \(1.12+(0.84)(3)=3.64\) concentration units.
Now fit the line again after excluding the flagged sample, solely to see how much that case affects the result. For the remaining four samples, \(\bar{x}=1.5\) hours and \(\bar{y}=9.48/4=2.37\) concentration units. Their centered \(x\)-values are \(-1.5,-0.5,0.5,1.5\), so \(S_{xx}=2.25+0.25+0.25+2.25=5\). Their centered \(y\)-values are \(-0.89,-0.53,-0.37,1.79\). Hence:
The case-excluded slope is \(4.10/5=0.82\) concentration units per hour, and its intercept is \(2.37-(0.82)(1.5)=1.14\) concentration units. The case-excluded prediction at \(x=3\) is \(1.14+(0.82)(3)=3.60\) concentration units. Relative to the primary fit, the slope changes by \(|0.84-0.82|=0.02\) concentration units per hour, and the prediction changes by \(|3.64-3.60|=0.04\) concentration units.
Here the comparison suggests little change in these two summaries for this particular data set. It does not show that the flagged sample is erroneous or that the full-data fit is universally appropriate. The primary analysis still retains the valid sample.
Worked Example: Translating Fit Differences Into Context
Consider a separate hypothetical audit of response \(y\), measured in response units, against predictor \(x\), measured in predictor units. The all-valid-case fit has line \(\hat{y}=1.12+0.84x\). A justified case-excluded sensitivity fit has line \(\hat{y}=0.90+0.90x\). These are alternative fit summaries for this example; the purpose is to calculate and interpret their differences, not to infer a deletion rule from the coefficients.
The slopes differ by:
Thus the estimated increase in response per one predictor unit is 0.06 response units per predictor unit smaller in the all-case fit than in the alternative fit. The slopes are both positive, so the estimated direction of association does not change in this comparison.
At \(x=3\) predictor units, the all-case prediction is \(1.12+(0.84)(3)=3.64\) response units. The alternative prediction is \(0.90+(0.90)(3)=3.60\) response units. Their absolute difference is:
This comparison shows why coefficients and predictions should both be considered. Slopes describe change per predictor unit, while a prediction is tied to a specified \(x\)-value. Intercepts can also differ, and two lines with different slopes can give similar predictions at one particular value. A report should therefore not describe the whole analysis as “unchanged” merely because one selected prediction differs little.
Whether a difference of 0.04 response units is practically small depends on the response scale and the application. The calculation alone does not answer that question. A careful conclusion states the numerical changes and then relates them to a meaningful scale or decision threshold, if one has been established.
Worked Example: Comparing a Multiple-Regression Coefficient
Suppose a hypothetical analysis estimates mean weekly response in units from a predictor measuring weekly exposure in hours, while adjusting for age. The primary fit includes all valid records and estimates the exposure coefficient as \(2.4\) response units per hour. A fit excluding one flagged but valid record estimates it as \(1.8\) response units per hour. The estimated adjusted association remains positive, but its magnitude changes.
The absolute coefficient change is:
The relative change, using the primary estimate as the reference, is \(0.6/2.4=0.25\), or 25%. In context, the case-excluded estimate is 25% smaller than the all-case estimate. This percentage is a descriptive comparison, not an error rate and not evidence that either estimate is the true value.
A complete interpretation would also check whether the change affects a substantive conclusion. For example, if a decision depended on whether the adjusted association exceeded 2 response units per exposure hour, the estimates fall on opposite sides of that benchmark. That makes the conclusion sensitive to the case-exclusion choice. The appropriate report would show both estimates, identify the case and reason for the sensitivity fit, and retain the all-valid-case result as primary unless a separate data-quality or target-population justification supports exclusion.
A Short Justification for Comparing Predictions
A slope change is not necessarily the same as a prediction change at every \(x\). Let the primary line be \(b_0+b_1x\) and the alternative line be \(b_0^*+b_1^*x\). Subtracting the two fitted values at a specified \(x\) gives:
This identity follows by subtracting \(b_0+b_1x\) from \(b_0^*+b_1^*x\) and grouping the intercept and slope terms. It shows that the intercept and slope changes can reinforce or offset each other at a given predictor value. At another \(x\), their combined effect may differ. Therefore, prediction comparisons should name the predictor values and should focus on values relevant to the analysis, not on an arbitrary point outside the data range.
How to Report Sensitivity Clearly
A transparent report identifies the primary model and gives its estimate, then states the precise change made for each sensitivity fit. For a case-excluded fit, identify the case by a non-identifying label and explain why it was examined. Do not call a valid case “bad data” merely because it affects the fit. If the observation is confirmed erroneous, describe the evidence and the correction or exclusion rule.
Compare quantities that address the question: coefficient estimates with units, predictions at relevant predictor values, and uncertainty summaries when inference is part of the goal. If a confidence interval or test conclusion changes, report that change rather than choosing whichever version is more convenient. Different fits do not automatically provide independent evidence, and a sensitivity comparison does not assign a probability that one model is correct.
A practical sequence is:
Check measurement, transcription, units, eligibility, and relevance to the stated target.
Use all observations judged valid for the analysis and state the model and target clearly.
For example, refit without a flagged valid case as a diagnostic comparison, and label that change explicitly.
Calculate changes in relevant coefficients or predictions, give units, and judge their practical importance.
Explain what changed, what did not, and which analysis remains primary and why.
Common Mistake: Treating a Flag as Permission to Delete
Incorrect reasoning: “This case was flagged as influential, so it must be an outlier error; I will delete it and report the cleaner fit.” A diagnostic identifies a case that merits attention; it does not establish a data error. Deletion can change the question being answered and can conceal valid variation.
Correction: Verify whether the case is valid and relevant to the target. Retain it in the primary analysis if it is. Then use an explicitly labeled alternative fit to assess sensitivity, report the numerical changes in context, and explain any independent reason for a correction or exclusion. A fit that looks more stable after deletion is not, by itself, justification for deleting the observation.
Check Your Understanding
Try these questions before moving on.
- A valid observation is flagged as influential. What does the flag establish, and what does it not establish?
- Why should a case-excluded fit generally be labeled as a sensitivity analysis when the excluded case is valid?
- Two fitted lines have slopes 1.5 and 1.2 response units per predictor unit. Calculate the absolute slope change and state its units.
- Explain why comparing predictions at a named, relevant predictor value can add information beyond comparing slopes.
- What should a report include if a substantive conclusion changes between the primary fit and a justified sensitivity fit?