Why Diagnostic Findings Need More Than a Verdict
A report that says “the diagnostics were satisfactory” leaves readers unable to tell what was examined or what the plots showed. A useful account gives three pieces of information: the observed pattern, its possible implication for the fitted model, and the follow-up analysis performed. This is not a claim that plots prove a model is right or wrong. It is a concise record of evidence that helps readers judge how the reported estimates should be interpreted.
For example, a curved pattern in a residual plot may indicate that the fitted mean relationship does not capture curvature. A widening spread may suggest that the error variance is not constant. Neither observation, by itself, identifies the one correct replacement model. The report should say what was seen, explain the concern in appropriately qualified language, and state what was done in response.
Model and Diagnostic Terms
Consider a regression model for an outcome \(Y_i\) and predictors \(x_{i1}, \ldots, x_{ik}\). Write the fitted value for observation \(i\) as \(\hat{y}_i\), and its residual as \(e_i = y_i - \hat{y}_i\). A residual is the observed outcome minus the outcome predicted by the fitted model.
A residual-versus-fitted plot places residuals against fitted values. A residual-versus-predictor plot places them against a predictor. If the mean form and variance are adequate, these plots should not show a systematic pattern such as a curve or a steadily widening band. A normal quantile plot compares the residual distribution with what would be expected under normality; pronounced departures, especially in the tails, may warrant closer attention when normal-theory inference is important. These are visual checks, not automatic pass-or-fail tests.
Two terms help describe cases that merit attention. Leverage measures how unusual an observation’s predictor values are relative to the other observations. A large residual means the observed outcome is far from the fitted value. An influential case is one whose inclusion has a substantial effect on fitted results, such as coefficients or fitted values. A case can have high leverage without a large residual, or a large residual without being influential.
Cook’s distance is one summary of influence. For a model with \(p\) estimated coefficients, including the intercept, and mean squared error \(MSE\), the Cook’s distance for observation \(i\) can be written as:
Here \(h_{ii}\) is the observation’s leverage. Cook’s distance summarizes how much the fitted values would change if that observation were omitted. It is a measure for locating cases to examine, not a rule that decides whether a case should be removed. There is no single cutoff that is appropriate in every data set and model.
How to Write a Diagnostic Finding
A clear report separates what was observed from what it might mean. “Residual spread increased with fitted values” describes a plot; “the error variance may not be constant” states a possible implication. Then identify the follow-up: for example, checking a record against its source, fitting a prespecified alternative mean form, or using a standard error method appropriate to nonconstant variance. State what the follow-up showed and which results are reported.
Specify which plot or influence measure was examined and describe the visible feature, rather than calling it simply “good” or “bad.”
Connect the pattern to a model concern, using measured language such as “suggested,” “raised concern about,” or “was consistent with.”
Say what was checked or changed, what it showed, and whether the main interpretation or its uncertainty requires qualification.
Worked Example: Curvature in a Residual Plot
Suppose a hypothetical model uses practice hours to predict a test score in points for 48 students. A residual-versus-practice-hours plot shows mostly negative residuals at low and high practice hours and mostly positive residuals in the middle. Describe the finding and a follow-up.
Observed pattern. The residuals form a curved pattern rather than a roughly patternless band around zero. This description refers to the plot; it does not yet claim that the model is incorrect.
Possible implication. The fitted linear mean relationship may not represent the relationship between practice hours and test score adequately. A systematic pattern in residuals is a reason to investigate the mean form.
Follow-up and report. In this hypothetical analysis, the analyst fitted a prespecified model that also included a squared practice-hours term and examined its residual plot. The pronounced curve was no longer apparent in that plot. A suitable report is: “The residuals from the linear model showed curvature across practice hours, suggesting that a linear mean form might be inadequate. After adding a squared practice-hours term, the residual plot showed no comparable curvature; estimates from the quadratic model are reported.”
This wording identifies the original pattern, the concern it raised, and the follow-up. It does not say that the second model is true merely because one plot looked less patterned; other assumptions and the scientific context still matter.
Worked Example: Increasing Residual Spread
A hypothetical regression predicts exam score in points from weekly study hours for 52 students. In the residual-versus-fitted plot, the vertical spread of residuals is narrow at lower fitted scores and wider at higher fitted scores. The estimated study-hours coefficient is 2.40 points per hour, with a conventional standard error of 0.62 points per hour.
Observed pattern and possible implication. The residual spread increases with fitted score, which raises concern that the error variance is not constant. The plot does not establish why the spread changes, and it does not show whether observations are independent.
Follow-up and report. In this hypothetical analysis, the analyst calculated heteroskedasticity-consistent standard errors as a follow-up. The coefficient remained 2.40 points per hour, while its standard error was 0.88 points per hour. The coefficient describes the same fitted change in predicted score per additional study hour; the larger standard error indicates greater uncertainty under this follow-up calculation. A report could say: “The residual spread increased with fitted score, raising concern about constant error variance. Heteroskedasticity-consistent standard errors were therefore used; the estimated coefficient for study hours was 2.40 points per hour (SE 0.88).”
This follow-up addresses uncertainty calculations under the variance concern; it does not remove the observed pattern or repair a misspecified mean relationship. The diagnostic finding and the method used to report uncertainty should both be stated.
Reporting Unusual and Influential Cases
When describing a potentially influential observation, report enough information for readers to understand why it was examined. Depending on the analysis, that may include its leverage, residual, Cook’s distance, or the results of checking its record. A large value on one diagnostic measure is a signal to investigate, not evidence of misconduct, measurement error, or a reason for automatic exclusion.
If an observation is valid and belongs to the intended analysis population, deleting it solely because it changes a result can distort the analysis. If a record is found to be erroneous, explain the error and how it was handled. If it is valid, say whether it was retained and what consequence the diagnostic check had for the interpretation. Keep the wording proportionate: an influential case can make a fitted result more dependent on that observation, but it does not by itself show that the result is unusable.
Worked Example: A Case With High Leverage and Cook’s Distance
Consider a hypothetical house-price model with 36 sales, three estimated coefficients including the intercept, and outcome measured in thousands of dollars. One sale has residual \(e_i=8\) thousand dollars, leverage \(h_{ii}=0.30\), and the model’s \(MSE=16\) in squared thousands of dollars. Calculate and interpret its Cook’s distance.
Substituting into the formula gives:
The sale has both relatively high leverage and a residual of 8 thousand dollars, and its Cook’s distance indicates that it merits examination for influence. The value \(0.816\) is not a universal deletion threshold and does not, on its own, establish that the sale is invalid.
In this hypothetical example, the analyst checked the sale against the source record and confirmed that the price and predictor values had been transcribed correctly. The observation was retained. A transparent report could read: “One sale had leverage 0.30, a residual of 8 thousand dollars, and Cook’s distance 0.816. We verified its recorded values against the source and retained the valid observation; the model’s fitted results should be interpreted with awareness that this case has notable influence.”
The report gives the observed diagnostics, the data check, and the decision. It does not call the case an outlier as a substitute for explaining what was unusual, and it does not imply that a large Cook’s distance proves the model or observation is wrong.
Common Mistakes and a Reporting Checklist
A common mistake is to write, “All assumptions were met because the diagnostic plots looked fine.” Plots can reveal some patterns, but they cannot establish independence from an observational design, prove normality, or guarantee correct model specification. A better statement names the plots examined and describes the absence of an obvious pattern, with an appropriately limited conclusion.
Another mistake is to report an influence flag without saying what happened next. “One observation exceeded the Cook’s distance cutoff” is incomplete if the cutoff is not justified and the reader cannot learn whether the record was checked, retained, or otherwise addressed. Report the measure and its context, the follow-up, and its consequence. Avoid presenting a diagnostic result as a formal hypothesis test unless a specific test was conducted and its assumptions and result are reported.
- Be specific: name the plot or diagnostic measure and describe the pattern or case.
- Be cautious: distinguish a possible implication from a demonstrated failure.
- Be transparent: state the follow-up performed and what it showed.
- Be contextual: explain how the finding affects interpretation of estimates, uncertainty, or fitted results.
- Do not automate judgment: a plot pattern or influence statistic alone does not prescribe a model change or case deletion.
Check Your Understanding
For each question, focus on separating the observed evidence, its possible implication, and the follow-up.
- A residual-versus-fitted plot shows a fan-shaped spread. What is the observation, what model concern might it raise, and what additional information should a report include?
- Why does high leverage not necessarily mean that an observation is influential?
- In a hypothetical model with \(p=2\), \(MSE=9\), residual \(e_i=3\), and leverage \(h_{ii}=0.20\), calculate Cook’s distance using the formula in this tutorial.
- A case has a large Cook’s distance but its data are verified as correct. Give one reason not to delete it automatically.
- Rewrite “The diagnostics were fine” as a more informative statement for a report that examined residual-versus-fitted and normal quantile plots and saw no obvious patterns.