When a Strong Ranking Is Not Enough
A model-selection criterion answers a relative question: which of the models under consideration has the preferred tradeoff between fit and complexity? It does not answer whether any model in that set adequately describes the data. This distinction matters when the numerically preferred model has residuals that retain a systematic pattern.
For regression, residual diagnostics help assess whether the fitted mean structure is reasonable and whether the errors behave as assumed. A smooth curve in residuals against fitted values, for example, can indicate that the candidate models have not captured a nonlinear mean relationship. If an information criterion ranks one of those models first, the ranking is still a valid comparison within the specified set, but it is not a reason to ignore the diagnostic evidence.
What the Score Does—and Does Not—Measure
For ordinary least squares regression with a Gaussian error model, one commonly used AIC-equivalent score is
where \(n\) is the number of observations, \(\mathrm{RSS}\) is the residual sum of squares, and \(p\) is the number of estimated regression coefficients in the mean model, including the intercept. A smaller score is preferred. This expression omits terms common to all candidates in the comparison. It is appropriate to compare scores only when the candidates use the same response values and the same observations, with comparable likelihood assumptions. If candidate models are fit on different response scales, this score alone does not provide a direct comparison.
The likelihood basis explains why fit and complexity both appear. Suppose the model has independent errors with mean zero, common variance \(\sigma^2\), and normally distributed errors, and its conditional mean is linear in the included coefficients. The errors are assumed to have constant variance, and the design matrix must have full column rank for the usual coefficient estimates to be identifiable. Under this model, the least-squares estimates maximize the Gaussian likelihood for the regression coefficients. The maximum-likelihood estimate of the error variance is \(\widehat{\sigma}^2=\mathrm{RSS}/n\).
There are \(p+1\) estimated parameters in the full likelihood if the variance parameter is counted as well. The Akaike information criterion is \(-2\) times the maximized log-likelihood plus twice the number of estimated parameters. Substituting the Gaussian likelihood gives a term involving \(n\log(\mathrm{RSS}/n)\), a penalty for the regression coefficients, and terms common to all candidates; the variance-parameter penalty is common too. Removing those common terms yields \(Q\) above. The score therefore rewards smaller residual variation but penalizes adding coefficients.
That penalty is not a diagnostic test. The criterion does not separately penalize a curved residual pattern, a sequence of same-sign residuals, or a small group of unusual residuals. Such features can still matter: they may show that the model assumptions or mean structure are inadequate. A smaller \(Q\) is evidence of a preferred tradeoff within the candidate set, not proof that the model is suitable for interpretation or prediction.
A Diagnostic-Led Decision Process
Check that candidates use the same response, observations, and score definition. A ranking is not meaningful if the inputs to the comparison are not comparable.
Calculate or read the scores accurately. Identify the best score, but describe it as the best among the candidates evaluated.
Look for a systematic feature that is larger or more organized than random scatter. Interpret it in the context of the fitted response and predictors.
Translate a plausible diagnostic pattern into a specific modeling question, such as whether a nonlinear mean term is needed. Do not add terms indiscriminately.
Compare the revised candidates using a suitable criterion and inspect their diagnostics again. A lower revised score is useful, but the revised model must also address the original pattern without creating a new problem.
The process is not a rule that every small deviation requires another term. Residuals are noisy, and flexible models can chase chance variation. The concern is a coherent, substantial pattern with a plausible explanation. The response should be proportionate: identify what the pattern suggests, test a focused revision, and report any remaining limitation.
Worked Example: The Lowest Score Still Leaves Curvature
A hypothetical insurer models claim cost using the same 20 policies and the same dollar response in three candidate regressions. Model A has \(p=2\) coefficients and \(\mathrm{RSS}=80\). Model B has \(p=3\) coefficients and \(\mathrm{RSS}=50\). Model C adds a further mean term to Model B, has \(p=4\), and \(\mathrm{RSS}=40\). The residual-versus-fitted plot for Model B shows a clear U-shaped pattern. Compare the scores and recommend a next step.
Method. Use the AIC-equivalent score \(Q=n\log(\mathrm{RSS}/n)+2p\). Because all three models use the same observations and response, their scores are comparable.
Model A. Here \(\mathrm{RSS}/n=80/20=4\), and \(2p=4\). Therefore,
Model B. Here \(\mathrm{RSS}/n=50/20=2.5\), and \(2p=6\). Thus,
Model C. Here \(\mathrm{RSS}/n=40/20=2\), and \(2p=8\). Thus,
The criterion ranks C first, then B, then A. But the stated U-shape in Model B residuals is evidence that its mean structure may be missing a nonlinear feature. Model C has the lowest score, yet its numerical ranking alone does not establish that the curvature has been resolved. The defensible next step is to inspect Model C’s residual plot and, if the pattern remains, consider a focused mean-model revision such as a quadratic term in a relevant predictor. The revised model must be refit and checked, not accepted merely because it has more terms.
Check. The score decreases from \(31.725887\) to \(24.325815\) to \(21.862944\), consistent with the declining residual sums of squares overcoming each added coefficient penalty. The recommendation avoids treating that ranking as a residual diagnostic.
Common mistake avoided: Do not conclude that Model B is adequate because its score is lower than Model A’s. The U-shaped residual evidence calls for checking the candidate set, and Model C must also be diagnosed.
Worked Example: A Lower Score and a Stronger Curvature Signal
A hypothetical model of repair cost uses 40 service records and the same response scale for each candidate. Model A has \(p=2\) coefficients and \(\mathrm{RSS}=120\). Model B has \(p=3\) and \(\mathrm{RSS}=100\). Model B’s residuals against a key predictor show a repeatable curve, while Model A’s show no comparably clear pattern. A proposed model D adds a quadratic term to Model A, has \(p=3\), and reduces \(\mathrm{RSS}\) to 70. Compare the scores and state what the evidence supports.
For Model A, \(\mathrm{RSS}/n=120/40=3\), with penalty \(2p=4\):
For Model B, \(\mathrm{RSS}/n=100/40=2.5\), with penalty \(2p=6\):
For Model D, \(\mathrm{RSS}/n=70/40=1.75\), with penalty \(2p=6\):
The score ranks D ahead of B and A. More importantly, D was motivated by the curvature and its residual sum of squares is lower than A’s despite having one additional coefficient. The evidence supports revising the candidate set to include the quadratic mean term. Before recommending D, inspect its residuals to confirm whether it addresses the curve and check that the improvement is not driven by an isolated observation or another unexplained feature.
Check. D’s score is lower than A’s by about \(19.560\) points, while its coefficient penalty rises by \(2\). The calculation is consistent with a substantial reduction in \(\mathrm{RSS}/n\).
Common mistake avoided: Do not treat the improvement in the score as proof that the quadratic model is correct. The diagnostic pattern motivates the candidate; the revised residuals determine whether the specific concern appears resolved.
Worked Example: An Influential Observation Is a Lead, Not an Automatic Deletion
A hypothetical agricultural study fits yield on two predictors using 30 plots. Model A has \(p=2\) and \(\mathrm{RSS}=90\), and its score is calculated using all plots. A diagnostic review identifies one plot with unusually high leverage and a large residual. Model B adds a scientifically plausible predictor, has \(p=3\), and \(\mathrm{RSS}=70\), also using all 30 plots. Calculate the scores and explain what should follow.
For Model A, \(\mathrm{RSS}/n=90/30=3\), and \(2p=4\):
For Model B, \(\mathrm{RSS}/n=70/30=7/3\), and \(2p=6\):
Model B has the lower score, but the influential plot remains a diagnostic concern. First verify the observation and its measurement, then assess whether the plot is part of the target population and whether the model form is appropriate for it. A case-deletion comparison can show sensitivity, but a large change after deletion is not, by itself, a justification for excluding a valid observation. If a justified revised model is considered, fit it to the stated analysis data and compare its diagnostics and criterion on a consistent basis.
Check. The scores use all 30 observations for both candidates, so their comparison is internally consistent. The leverage finding does not change the arithmetic; it changes how cautiously the fitted models should be interpreted.
Common mistake avoided: Do not remove an observation simply to improve residual plots or the selection score. Investigate its validity and influence, and explain any exclusion on substantive grounds.
Common Traps in Diagnostic-Led Selection
- Calling a relative winner adequate. “Lowest among these candidates” is a limited conclusion. If residuals retain a clear pattern, state that limitation and explain what model feature may be missing.
- Adding terms without a diagnostic rationale. A more complex candidate can reduce \(\mathrm{RSS}\) simply by fitting the observed sample more closely. A focused revision should connect to a feature the residual evidence suggests.
- Overreacting to random scatter. The goal is not to make every residual exactly zero or every plot look perfectly patternless. The concern is a systematic, interpretable structure, not ordinary sampling variation.
- Comparing unlike fits. The displayed score does not justify direct comparison if candidates use different response scales or different observations. Establish comparability before ranking.
- Deleting a point to silence a diagnostic. An unusual observation may be valid and important. Investigate it, assess sensitivity, and avoid treating deletion as a routine model-selection step.
- Stopping after a successful revision. A term motivated by one plot can reveal another issue or fail to address the original one. Recheck residuals after refitting.
Check Your Understanding
For each item, assume the stated candidates use the same response and observations unless the problem says otherwise. Use \(Q=n\log(\mathrm{RSS}/n)+2p\) and round scores to three decimal places.
- A model has \(n=30\), \(p=2\), and \(\mathrm{RSS}=90\). Its residual plot shows a clear curve. Calculate its score and name a reasonable next step. (Answer: \(Q=36.958\); investigate a plausible nonlinear mean term and reassess the residuals.)
- A model with \(n=25\), \(p=2\), and \(\mathrm{RSS}=52.5\) has residuals that alternate irregularly around zero with no clear pattern. Calculate its score. Does the score alone establish that the model is adequate? (Answer: \(Q=25\log(2.1)+4=22.548\); no, the score is a relative ranking tool, not an adequacy test.)
- For \(n=50\), Model A has \(p=3\), \(\mathrm{RSS}=150\); Model B has \(p=4\), \(\mathrm{RSS}=120\). Model B has a funnel-shaped residual pattern. Calculate both scores and give the cautious conclusion. (Answer: \(Q_A=60.931\), \(Q_B=51.773\); B ranks better, but the funnel pattern requires investigating the variance assumptions before treating B as adequate.)
- For \(n=24\), a linear model has \(p=2\), \(\mathrm{RSS}=72\). A candidate adding a quadratic term has \(p=3\), \(\mathrm{RSS}=48\), and removes the visible residual curve. Calculate both scores and state the next check. (Answer: linear \(Q=30.367\), quadratic \(Q=22.636\); inspect the revised model’s residuals for remaining structure and other diagnostic concerns.)
- Two models have different residual sums of squares, but one is fit on 28 observations and the other on 30 because missing responses were handled differently. Can their displayed \(Q\) scores be treated as a direct ranking? (Answer: not without addressing the different analysis samples; compare candidates using a consistent response and observation set, or justify another suitable comparison.)