Reporting Influence Diagnostics

Why Reporting Influence Diagnostics Requires More Than a List of Flags

Influence diagnostics can identify cases whose removal would noticeably change a fitted regression. A report should make that information understandable without implying that an unusual case is automatically wrong or that a particular cutoff is a formal test. Readers need to know which measures were examined, how potential flags were screened, what investigation followed, and what changed in any justified sensitivity analysis.

The central reporting principle is to separate three decisions: screening identifies cases for attention; investigation checks their data and context; and sensitivity analysis compares fits under a defensible alternative analysis. A diagnostic flag is a prompt to investigate, not a verdict about a data point.

State the Model and Define the Reported Quantities

Suppose a regression has $n$ observations and $p$ fitted coefficients, counting the intercept. For a multiple linear regression with $q$ predictors, $p=q+1$. A standard model is

Y_i=\beta_0+\beta_1x_{i1}+\cdots+\beta_qx_{iq}+\varepsilon_i,\qquad i=1,\ldots,n.

For the usual least-squares diagnostic interpretation, the model should have a correctly specified conditional mean, independent errors with mean zero and constant variance, and a full-rank predictor matrix. Normal errors are additionally used for exact small-sample $t$- and $F$-inference, but influence screening itself is descriptive and does not require a p-value. The report should name the model and outcome so that the diagnostics have context.

Definition: An influence diagnostic report identifies the measures and screening criteria used, describes how flagged observations were checked, and reports any justified sensitivity analyses. A flag means that a case meets a selected screening criterion; it does not establish that the case is erroneous, influential in every relevant sense, or appropriate to exclude.

Common measures include leverage $h_{ii}$, which describes how unusual an observation’s predictor values are; a studentized residual, which describes the discrepancy between observed and fitted response; Cook’s distance $D_i$, which summarizes the change in the fitted regression when case $i$ is removed; DFFITS, which summarizes the case-deletion change in its fitted value; and DFBETAS, which measure case-deletion changes in individual coefficients. A covariance ratio compares the estimated coefficient-uncertainty covariance after deleting a case with that from the full fit. These measures answer related but distinct questions. A report should not call them interchangeable.

Screening conventions are often expressed as rules of thumb. For example, commonly used candidate cutoffs are $4/n$ for Cook’s distance, $2p/n$ for leverage, $2\sqrt{p/n}$ for the absolute value of DFFITS, and $2/\sqrt{n}$ for the absolute value of a DFBETAS. These are conventions for locating cases to inspect, not universal boundaries between acceptable and unacceptable observations. State the exact rule used, including whether a criterion is applied to an absolute value.

Make the Screening Stage Reproducible

A concise report gives the sample size, the number of fitted coefficients, each diagnostic examined, and the screening rule. When several measures are used, state whether a case was flagged by any one measure or only by a specified combination. This matters: “cases exceeding the Cook’s distance cutoff were examined” describes a different screening set from “cases exceeding either the Cook’s distance or leverage cutoff were examined.”

When a cutoff is a heuristic, call it a screening criterion or rule of thumb, not a significance threshold. Diagnostic values do not by themselves supply the probability that a case is an error or the probability that an observation should be removed. If many diagnostics are inspected, the report should also avoid suggesting that each flag is independent confirmation: measures can flag the same case because they respond to overlapping features of the fitted model.

Worked Example: Reporting the Screening Rules

A hypothetical analysis fits a linear regression to $n=50$ observations with four predictors and an intercept, so $p=5$. The analyst screens using Cook’s distance, leverage, DFFITS, and DFBETAS. Calculate the stated rules of thumb and identify the cases flagged in a fictional diagnostic summary.

The screening cutoffs are

\frac{4}{n}=\frac{4}{50}=0.0800,\qquad \frac{2p}{n}=\frac{2(5)}{50}=0.200,

2\sqrt{\frac{p}{n}}=2\sqrt{\frac{5}{50}} =2\sqrt{0.100}=0.632, \qquad \frac{2}{\sqrt{n}}=\frac{2}{\sqrt{50}}=0.283.

Suppose case 12 has $D_{12}=0.11$, $h_{12,12}=0.22$, $|\mathrm{DFFITS}_{12}|=0.67$, and $|\mathrm{DFBETAS}_{12}|=0.31$ for the coefficient of interest. It exceeds all four stated cutoffs. Suppose case 31 has $D_{31}=0.09$ and $h_{31,31}=0.24$, but its DFFITS and DFBETAS do not exceed their cutoffs. It is flagged by Cook’s distance and leverage, not by those two other measures. A clear report preserves these distinctions rather than describing both cases simply as “outliers.”

A concise screening statement could read: “In the 50-observation model with five fitted coefficients, cases were screened using Cook’s distance greater than $4/n=0.080$, leverage greater than $2p/n=0.200$, absolute DFFITS greater than $2\sqrt{p/n}=0.632$, and absolute DFBETAS greater than $2/\sqrt n=0.283$. Case 12 exceeded all four rules; case 31 exceeded the Cook’s distance and leverage rules.” The statement identifies the rules and what they found without presenting the cutoffs as formal tests.

Describe Investigation, Not Just Diagnostic Output

After screening, investigate flagged records using the information available about data collection and the subject matter. Checks may include comparing a value with its source record, verifying units and coding, checking whether values were transcribed correctly, and determining whether the observation belongs to the target population. Report what was checked and the outcome at an appropriate level of detail. Do not disclose identifying information when doing so would violate privacy.

If a value is confirmed to be a data-entry or measurement error, correction or exclusion may be warranted under a stated and consistently applied data-handling rule. If the value is verified and represents a valid member of the target population, its statistical unusualness alone is not a reason to remove it. A valid extreme observation may be important to the population relationship the analysis is intended to describe.

Worked Example: Reporting a Valid High-Leverage Case

In a hypothetical study of response time in seconds as a function of a measured exposure, case 31 was flagged for leverage and Cook’s distance. The analyst checks the original form, confirms the exposure unit and response-time transcription, and verifies that the participant meets the study’s inclusion criteria. The exposure is high relative to most observations, but it is within the range allowed by the study protocol. Its residual is small.

The appropriate conclusion is not that case 31 is an error. The case has unusual predictor values, but its observed response is close to the value fitted by the model. That combination can produce high leverage without a large residual. Since the source record and eligibility are verified, the analyst retains the observation in the primary analysis.

A report might say: “Case 31 met the leverage and Cook’s distance screening criteria. Source records, units, and eligibility were checked and verified; the exposure value was within the protocol’s permitted range. The case was retained in the primary model.” If a sensitivity analysis excluding case 31 is also justified and conducted, its results should be reported separately as a check on how that decision affects the estimates. The diagnostic flag alone does not settle the decision.

Report Sensitivity Analyses in Context

A case-deletion sensitivity analysis refits the same model after removing a specified observation, then compares the results with the full-data fit. It asks how dependent the reported results are on that case under this particular alternative analysis. It does not establish that the deleted fit is preferable, nor does it replace the primary analysis. State which case was omitted, why that analysis was considered, what remained unchanged in the model, and which quantities changed.

For a coefficient, report both estimates, their units, and their difference. If a relative change is useful and the full-model estimate is nonzero, calculate it as the change divided by the full-model estimate. Be careful with signs: a negative change can indicate a smaller estimate, while a positive percentage is sometimes used to describe the magnitude of a decrease.

Justification: If the full-model estimate of a coefficient is $\hat\beta_{\text{full}}$ and its case-deletion estimate is $\hat\beta_{(-i)}$, the signed change is $$ \Delta_i=\hat\beta_{(-i)}-\hat\beta_{\text{full}}. $$ When $\hat\beta_{\text{full}}\ne 0$, the signed relative change is $100\Delta_i/\hat\beta_{\text{full}}$ percent. This comparison is meaningful only with the coefficient’s scale and units stated; it is not a test of whether the two estimates differ statistically.

A useful sensitivity summary may also compare standard errors, fitted values at a substantively relevant predictor profile, or other estimates central to the research question. If uncertainty changes, describe it as a change in estimated uncertainty, not proof that one fit is more correct. A sensitivity analysis should not be presented as a search for whichever model gives the preferred result.

Worked Example: Summarizing a Case-Deletion Change

A hypothetical response is a test score in points, and one predictor is dose in milligrams. A model uses $n=40$ observations, three predictors, and an intercept ($p=4$). Case 17 is flagged by the chosen influence screens. The full-data slope for dose is $0.42$ score points per milligram, with standard error $0.16$. After deleting case 17, the same model has a dose slope of $0.28$ score points per milligram, with standard error $0.14$.

The signed change in the dose estimate is

\Delta_{17}=0.28-0.42=-0.14 \quad\text{score points per milligram}.

The signed relative change is

100\left(\frac{-0.14}{0.42}\right)=-33.3\%.

Thus, the estimated dose slope decreases by $0.14$ score points per milligram, or $33.3\%$ of its full-model estimate, when case 17 is omitted. The standard error also decreases from $0.16$ to $0.14$ score points per milligram, a change of $0.02$. The deletion estimate remains positive, but its magnitude is appreciably smaller. These facts support describing the estimated slope as sensitive to case 17; they do not establish which estimate is correct.

A suitable report would give the full and case-deletion estimates, identify the case and reason for the sensitivity analysis, and say whether the main interpretation changes. If the case was valid, the full-data model remains the primary analysis unless a separate, defensible reason supports exclusion. The deletion fit is a sensitivity check, not a replacement chosen because its result is more convenient.

Common Mistakes and a Practical Reporting Sequence

A frequent mistake is to write, “The influential observation was removed because its Cook’s distance exceeded the cutoff.” This treats a screening convention as an exclusion rule. The correction is to report the flag, investigate the observation, give the reason for any alteration to the data, and show the effect of a justified alternative analysis. Another mistake is to say “the coefficient was unchanged” without giving enough precision or a scale to assess that claim. Report the estimates and their difference in context.

The following sequence makes a report both concise and auditable:

Identify the fitted model.
Name the outcome, predictors, sample size, and number of fitted coefficients. State any model features needed to understand the analysis.

Give the diagnostics and screening rules.
Name each measure, provide the cutoff and its basis, and say whether cases were flagged by any rule or a specified combination.

Summarize investigation.
Describe the record checks and whether each flagged case was verified, corrected, excluded, or retained, with the reason.

Present sensitivity results.
For any defensible alternative fit, identify the change made and report relevant estimate changes with units and enough precision.

State the implication with restraint.
Describe which conclusions are robust or sensitive to the case; do not treat a diagnostic flag as proof of error or causation.

A good report is selective rather than a dump of every software output. Include the measures that support the screening decision and the sensitivity results that bear on the analysis’s main claims. Keep the full-data analysis, decisions about records, and alternative fits clearly distinguished so readers can see what was observed, what was changed, and why.

Key takeaway: Report influence diagnostics as a transparent sequence: measures and screening criteria, investigation of flagged cases, and the effect of any justified sensitivity analysis. A flag calls for attention; it does not by itself justify deleting an observation.

Check Your Understanding

Answer each question by distinguishing screening, investigation, and sensitivity analysis.

Why should a report call a Cook’s distance cutoff a screening rule of thumb rather than a significance threshold?
A case exceeds a leverage rule but its source record is verified and it belongs to the target population. What should the analyst report, and does the flag alone justify exclusion?
For $n=80$ and $p=5$, calculate the rules of thumb $4/n$ and $2p/n$. What do these cutoffs mean?
A full-model slope is $1.20$ units of response per predictor unit, and a case-deletion slope is $0.90$. Find the signed change and signed relative change.
What information should accompany a case-deletion estimate so that readers can interpret the sensitivity analysis?