Tutorials › Regression Analysis (Undergraduate) › Case-Deletion Sensitivity Analysis

Home  /  Tutorials  /  Leverage and Influential Observations

Leverage and Influential Observations · Tutorial 563 of 991

Case-Deletion Sensitivity Analysis

Compare a regression analysis with and without selected cases, and distinguish changes in coefficients, fitted values, and uncertainty.

Intermediate 10 min read

What You'll Learn

  • Define a case-deleted regression fit and the conditions needed to compare it with a full-data fit.
  • Compare full-data and case-deleted coefficient estimates in their original units.
  • Assess whether fitted mean responses at a fixed predictor profile change after deletion.
  • Compare standard errors and confidence intervals without confusing precision with effect size.
  • Summarize which conclusions are stable and which depend on a particular case.
  • Use one-case-at-a-time deletion as a sensitivity check, not as automatic grounds for removing data.

Why Compare a Fit With and Without a Case?

A regression model summarizes a relationship across the observations used to fit it. A particular observation can sometimes have a noticeable effect on that summary. A useful follow-up is to fit the same model again after removing one case, then compare the two results. This asks a practical question: Would the interpretation or conclusion be materially different if this one observation were not included?

This comparison is a sensitivity analysis. It does not, by itself, establish that an observation is incorrect or that it should be excluded. The full-data fit remains the analysis of the stated data set; a case-deleted fit shows how dependent that analysis is on the case examined. The reason for examining a case should come from a stated concern or a diagnostic review, not from searching for a deletion that produces a preferred result.

Definition and Conditions for a Meaningful Comparison

Consider a multiple linear regression with \(n\) observations and \(q\) fitted coefficients, including the intercept. For observation \(i\), let \(y_i\) be the response and let \(\mathbf{x}_i\) be its predictor vector, including the intercept term. The model is

$$ y_i=\mathbf{x}_i^\mathsf{T}\boldsymbol{\beta}+\varepsilon_i,\qquad i=1,\ldots,n. $$

For the usual OLS comparisons, the predictor matrix must have full column rank. The conditional mean is represented by the linear model, and errors are independent with mean zero and constant variance. Normal errors additionally support exact small-sample \(t\)-based confidence intervals. These conditions are assumptions about the model and data-generating process; deleting a case does not make them automatically true.

Definition: The case-deleted fit for case \(i\) is the OLS fit obtained by removing observation \(i\), while retaining the same response, predictors, model form, and coding for all remaining observations. Write its coefficient estimate as \(\hat{\boldsymbol{\beta}}_{(-i)}\). A case-deletion sensitivity analysis compares selected estimates and uncertainty from this fit with those from the full-data fit, \(\hat{\boldsymbol{\beta}}\).

The comparison is most interpretable when the same model is estimable after deletion: the remaining predictor matrix must still have full column rank. With \(q\) coefficients, the residual degrees of freedom are \(n-q\) in the full fit and \(n-1-q\) in a case-deleted fit. If deleting a case makes a coefficient unidentifiable, then that deleted fit cannot support a like-for-like coefficient comparison.

Compare quantities that answer the scientific question. These may include selected coefficients in their original units, fitted mean responses at a fixed predictor profile, standard errors, confidence intervals, or a decision such as whether an interval includes zero. Keep predictor values and the target of the comparison fixed. A changed coefficient does not necessarily imply a changed prediction: several coefficient changes can offset one another at a particular predictor profile.

A Structured One-Case-at-a-Time Check

1
State the concern and the comparison.
Specify why the case is being examined and which estimates, fitted means, or intervals matter. Do not choose a case simply because its deletion gives a preferred answer.
2
Fit the stated model to all cases.
Record the full-data estimates, standard errors, and relevant fitted values or intervals, with units and the predictor profile made explicit.
3
Refit after removing only the case under review.
Keep the model form and variable definitions unchanged. Check that the reduced predictor matrix remains full rank and that the fitting assumptions remain reasonable for the comparison.
4
Compare size, uncertainty, and conclusions.
Describe what changed and what did not. A small p-value changing sides of a cutoff is not the only measure of sensitivity; the effect estimate, its units, and its interval also matter.

A useful numerical summary for a coefficient is the signed change \(\hat{\beta}_{j(-i)}-\hat{\beta}_j\), which preserves the coefficient's original units. For a fitted mean at predictor vector \(\mathbf{x}_0\), compare \(\mathbf{x}_0^\mathsf{T}\hat{\boldsymbol{\beta}}_{(-i)}\) with \(\mathbf{x}_0^\mathsf{T}\hat{\boldsymbol{\beta}}\). These are direct comparisons, not universal thresholds for declaring a fit “sensitive.” Whether a difference matters depends on the scale and the scientific question.

Worked Example: A Coefficient and Its Significance Conclusion

Suppose a hypothetical data set contains 30 independent facilities. The response is monthly energy use in kWh, and the predictors are an insulation score and building age. A linear conditional mean is considered plausible, the predictor matrix has full rank, and residual checks for this exercise show no major departure from constant variance or the assumed error model. Compare the insulation coefficient from the full fit with the fit deleting one case that was flagged for review.

The full fit estimates the insulation coefficient as \(2.40\) kWh per score point, with standard error \(0.80\) kWh per score point. It uses \(30-3=27\) residual degrees of freedom: there are three coefficients, counting the intercept. The test statistic for \(H_0:\beta_{\text{insulation}}=0\) is

$$ t=\frac{2.40-0}{0.80}=3.00. $$

For a two-sided test, this gives approximately \(p=0.0057\). In the case-deleted fit, the coefficient is \(1.75\) kWh per score point, with standard error \(0.78\), and there are \(29-3=26\) residual degrees of freedom. Its statistic is

$$ t=\frac{1.75-0}{0.78}=2.24. $$

The corresponding two-sided p-value is approximately \(0.034\). At \(\alpha=0.05\), both fits provide evidence that the coefficient differs from zero, but the estimated association is smaller after deletion: \(1.75-2.40=-0.65\) kWh per score point. The appropriate summary is that the direction and 5% significance conclusion are stable in this comparison, while the estimated magnitude is somewhat sensitive to the case. This is not evidence that insulation causes energy use to change; it is a comparison of adjusted regression associations.

Fitted Mean Responses Can Be More Stable Than Coefficients

A coefficient describes a predictor's modeled association while holding the other predictors fixed. A fitted mean response combines all the coefficients at a particular predictor profile. Consequently, coefficient changes may partly cancel when the fitted mean is calculated. The comparison must be made at the same \(\mathbf{x}_0\) in both fits.

Worked Example: Comparing a Fitted Mean at a Fixed Profile

In a hypothetical model of service waiting time, \(x_1\) is a workload score and \(x_2\) is staffing level. The response is measured in minutes. The model uses the same two predictors and an intercept in each fit. For illustration, the sites are independent, the linear mean structure is reasonable, residual spread is approximately constant, and both full and reduced predictor matrices have full rank.

At the profile \(x_1=4,\ x_2=3\), suppose the full-data coefficients are \((\hat{\beta}_0,\hat{\beta}_1,\hat{\beta}_2)=(12,1.8,-0.6)\). The fitted mean waiting time is

$$ \hat{y}_0=12+(1.8)(4)+(-0.6)(3)=12+7.2-1.8=17.4\text{ minutes}. $$

After deleting one reviewed case, suppose the coefficients are \((10.5,2.1,-0.4)\). At the same profile,

$$ \hat{y}_{0(-i)}=10.5+(2.1)(4)+(-0.4)(3)=10.5+8.4-1.2=17.7\text{ minutes}. $$

The intercept and both slopes changed, but the fitted mean at this profile changed by only \(17.7-17.4=0.30\) minutes. That difference may be small or important depending on the application; its size should be judged in context. It does not show that every fitted mean is equally stable. A different predictor profile could combine the changed coefficients differently.

Comparing Uncertainty, Not Just Point Estimates

For a specified predictor vector \(\mathbf{x}_0\), a confidence interval for the mean response concerns the conditional mean at that profile, not the response of one future individual. In a full-rank OLS model, its usual form is

$$ \mathbf{x}_0^\mathsf{T}\hat{\boldsymbol{\beta}} \ \mathbin{\pm}\ t_{1-\alpha/2,\ n-q}\,SE\!\left(\mathbf{x}_0^\mathsf{T}\hat{\boldsymbol{\beta}}\right). $$

The case-deleted interval uses the reduced fit's estimate, standard error, and \(n-1-q\) degrees of freedom. Its width can change because both the estimated error variance and the predictor geometry have changed. A wider interval is less precise; it does not by itself mean the estimated effect is larger. The usual \(t\)-based interpretation relies on the stated regression assumptions, including normal errors for exact finite-sample coverage.

Worked Example: Does an Interval-Based Conclusion Change?

Suppose a hypothetical model estimates mean recovery time at a fixed treatment-and-age profile. Recovery time is measured in days. The observations are independent, the linear model is adequately specified for this comparison, errors have constant variance and are normally distributed, and the design remains full rank after deleting the reviewed case. Assume \(q=3\) fitted coefficients and \(n=40\). The full fit gives a mean estimate of \(50.0\) days and \(SE=1.20\) days. The case-deleted fit gives \(50.4\) days and \(SE=1.75\) days.

For a 95% interval, the full fit has \(40-3=37\) degrees of freedom and \(t_{0.975,37}\approx2.026\). Its margin of error is

$$ (2.026)(1.20)=2.431\text{ days}, \qquad 50.0\mathbin{\pm}2.431=(47.57,\ 52.43)\text{ days}. $$

The reduced fit has \(39-3=36\) degrees of freedom, for which \(t_{0.975,36}\approx2.028\). Its margin of error and interval are

$$ (2.028)(1.75)=3.549\text{ days}, \qquad 50.4\mathbin{\pm}3.549=(46.85,\ 53.95)\text{ days}. $$

Both intervals lie above 45 days, so the conclusion that the mean at this profile exceeds 45 days is unchanged by this deletion under the stated interval procedure. The case-deleted estimate is only 0.4 days higher, but its interval is wider because its standard error is larger. These are 95% confidence intervals for a mean response; they are not intervals for an individual patient's recovery time and do not assign a 95% probability to the fixed mean being inside the realized interval.

How to Summarize Sensitivity

A clear summary separates three questions: did the estimate move, did its uncertainty change, and did the substantive conclusion change? One deletion may have little effect on a fitted mean but a noticeable effect on a coefficient, or it may leave point estimates similar while widening an interval. Report the actual comparison and its context rather than using a single label such as “influential” to stand in for all three.

Deletion is a diagnostic comparison, not a data-cleaning rule. If the case is valid, its information belongs to the stated analysis unless there is a defensible reason to define a different target population or model. If several cases are examined, make clear that each fit removes one case at a time; simultaneous deletion answers a different question. Repeatedly trying deletions until a desired conclusion appears is not a justified sensitivity analysis.

Common Mistake

  • Wrong reasoning: “The coefficient is no longer significant after deletion, so the case must be removed.” A significance cutoff alone does not establish that a case is erroneous or that its removal is justified.
  • Correction: Report the full-data and case-deleted estimates, uncertainty, and context. Investigate the case on substantive and data-quality grounds, and keep the primary analysis tied to the stated data unless a defensible exclusion rule applies.
  • Wrong reasoning: “The coefficients changed, so every prediction changed substantially.” Coefficients combine to form fitted values; changes can offset at some predictor profiles.
  • Correction: Compare fitted means at the same explicitly stated predictor profile, and compare uncertainty separately.
Key takeaway: Case-deletion sensitivity analysis asks how selected regression estimates, fitted means, uncertainty, and conclusions change when one case is omitted while the model is held fixed. A difference identifies dependence on that case; it does not by itself justify excluding the case.

Check Your Understanding

Use the definitions and comparison principles above to answer these questions.

  1. A model has 25 observations and 4 fitted coefficients, including the intercept. What are its residual degrees of freedom, and what are the degrees of freedom after deleting one case?
  2. Why should the full and case-deleted fitted means be compared at the same predictor profile?
  3. A coefficient changes from 3.0 to 2.6, but its standard error also changes. Why is comparing the coefficient estimates alone incomplete?
  4. What does it mean if a confidence interval for a mean response is wider after case deletion but its center changes very little?
  5. Why does a change in statistical significance after deleting a case not, by itself, justify removing that case?

Previous

Next

Back to Tutorials