Residual and Influence Diagnostics in Association Regression

Why Diagnose an Association Regression?

A covariate-adjusted association regression summarizes how a genetic dosage relates to a phenotype under a specified model. For a quantitative phenotype, a common starting point is a linear model; for a binary phenotype, it is a logistic model. In either case, a small standard error or a strong test statistic does not establish that the model’s functional form is adequate or that the numerical fit is trustworthy. Diagnostics help identify patterns the model has not captured and observations that may disproportionately affect its estimates.

Here, \(g_i\) denotes the genotype dosage for individual \(i\), and \(z_i\) denotes that individual’s vector of covariates. In an additive model, the genotype term treats dosage as a numerical predictor, so its coefficient describes the change in the modeled outcome per additional dosage unit, conditional on the included covariates. Diagnostics do not decide which covariates should be included, establish causation, or prove that a model is correct. They guide targeted checks of the model and the data.

Residuals in Linear Association Models

For a linear model with an intercept, genotype dosage, and covariates, write \(y_i=x_i^\mathsf{T}\beta+\varepsilon_i\), where \(x_i\) collects the predictors and \(\beta\) their coefficients. The fitted value is \(\hat{y}_i=x_i^\mathsf{T}\hat{\beta}\), and the raw residual is \(e_i=y_i-\hat{y}_i\). A residual plot displays these discrepancies against fitted values, genotype dosage, or a covariate.

The usual linear-model conditions for conventional model-based inference include a correctly specified conditional mean, independent errors with mean zero and constant variance, and (for exact small-sample \(t\)- and \(F\)-inference) normally distributed errors. Residual plots can reveal evidence against the mean-form or constant-variance conditions, but they cannot prove the conditions hold. In large association analyses, robust standard errors may address some variance misspecification; they do not repair an incorrectly specified conditional mean.

Look for systematic shape rather than expecting every residual to be zero. Curvature against a continuous covariate can suggest that its modeled effect is too simple. A funnel-shaped spread can suggest nonconstant variance. A pattern against genotype dosage can indicate that the assumed per-dosage linear effect is inadequate. These are reasons to examine a scientifically defensible alternative, such as a nonlinear covariate effect or a different genotype specification, rather than to add terms simply because a plot looks imperfect.

Leverage and Influence in Linear Models

In ordinary least squares, the hat matrix is \(H=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}\), where \(X\) is the design matrix containing all modeled predictors. Its diagonal element \(h_{ii}\), also written \(h_i\), is the leverage of observation \(i\). With \(p\) fitted coefficients, including the intercept, the leverages sum to \(p\), so their average is \(p/n\). An observation with unusual predictor values can have high leverage even when its residual is small.

Cook’s distance combines residual size and leverage to summarize how much the fitted regression would change if one observation were omitted. For the usual least-squares model, let \(\mathrm{MSE}=\sum_i e_i^2/(n-p)\). Then

Cook’s distance is a screening measure, not a universal test with a definitive cutoff. Rules such as \(4/n\) can flag observations for follow-up, but should not determine exclusion on their own. The substantive question is whether the observation is valid, whether the model represents it adequately, and whether the scientific conclusion is sensitive to its inclusion.

Why Cook’s Distance Has This Form

The formula can be justified from the change in least-squares coefficients after deleting observation \(i\). Let \(\hat{\beta}_{(-i)}\) denote the coefficient estimate with that observation omitted. The deletion identity is

where \(x_i\) is the column vector of predictors for that observation. This identity follows by expressing the deleted cross-product matrix as \(X^\mathsf{T}X-x_ix_i^\mathsf{T}\) and applying the rank-one matrix inverse formula. Cook’s distance scales the squared coefficient change by the model’s residual variance and the number of fitted coefficients:

Substituting the deletion identity into this expression gives the stated formula because \(x_i^\mathsf{T}(X^\mathsf{T}X)^{-1}x_i=h_i\). Thus a large residual increases Cook’s distance, and high leverage increases it further; neither feature alone determines influence.

Residuals and Leverage in Logistic Association Models

For a binary outcome \(y_i\in\{0,1\}\), a covariate-adjusted logistic model specifies \(\operatorname{logit}(\mu_i)=x_i^\mathsf{T}\beta\), where \(\mu_i=P(Y_i=1\mid x_i)\) and \(\operatorname{logit}(\mu_i)=\log\{\mu_i/(1-\mu_i)\}\). The genotype coefficient is on the log-odds scale. Logistic residuals are not interpreted as differences in the original outcome scale in the same way as linear-model residuals.

Two common residuals are the Pearson residual \(r_i=(y_i-\hat{\mu}_i)/\sqrt{\hat{\mu}_i(1-\hat{\mu}_i)}\) and the deviance residual. The latter is the signed square root of the observation’s contribution to the binomial deviance:

Terms of the form \(0\log(0)\) are defined as zero. Large residuals can identify observations whose outcomes are poorly predicted, but their magnitudes depend on the fitted probability. Logistic leverage is based on the weighted design matrix: with \(W\) diagonal and \(w_i=\hat{\mu}_i(1-\hat{\mu}_i)\), the weighted hat matrix is \(H_W=W^{1/2}X(X^\mathsf{T}WX)^{-1}X^\mathsf{T}W^{1/2}\). Its diagonal identifies observations with unusual influence on the weighted fit.

For logistic regression, influence can be assessed by refitting after omitting one observation and comparing the coefficient vectors. A clear distance measure is

Here \(X^\mathsf{T}WX\) is evaluated at the full-data fit. This scaled coefficient-change measure is dimensionless; it is a diagnostic definition, not a universal cutoff. Software may report other approximations or scalings, so identify the measure and its convention before comparing values.

Convergence Checks Are Part of Diagnosis

Residuals and influence measures are meaningful only when the fitting procedure has produced a usable estimate. In logistic regression, maximum-likelihood fitting is iterative. A software message that an algorithm stopped is not by itself enough to establish convergence. Review the convergence status, the size of the score vector (the gradient of the log-likelihood), whether the information matrix can be inverted, and whether coefficient estimates and standard errors are finite. If estimates change materially with starting values or numerical tolerances, the fit requires further investigation.

For example, a hypothetical logistic fit reaches its iteration limit with a genotype coefficient of \(0.24\), a score norm of \(0.08\), and no finite standard error reported. This is not a basis for interpreting the coefficient as a stable association estimate. Check scaling and the optimization output, and investigate sparse predictor combinations or separation. Earlier material on logistic separation and Firth regression addresses those specific issues. A converged fit is necessary for interpretation, but it does not guarantee correct model specification.

Common Pitfall: Treating a Diagnostic Flag as a Deletion Rule

A common error is to delete every observation with a large residual, high leverage, or Cook’s distance above a rule of thumb, then report the resulting estimate as the corrected answer. The reasoning is flawed: a diagnostic flag identifies a feature that merits investigation, not an invalid participant. Removing observations based only on their effect on the result can distort the analysis and conceal real heterogeneity.

Instead, verify the data and modeling assumptions, retain valid observations in the primary analysis unless a defensible exclusion criterion applies, and assess sensitivity with clearly described refits. Diagnostics also have limits: residuals cannot reveal every omitted confounder, and a well-behaved plot cannot prove that a genetic association is unbiased. They are tools for scrutinizing the fitted model, not substitutes for study design or scientific judgment.