Why Careful Comparisons Matter
An information criterion gives a numerical way to compare candidate models by balancing fit against a penalty for complexity. But the arithmetic is only meaningful when the scores are on a comparable basis. A score can look precise while answering the wrong question if the models use different observations, the calculation conventions are mixed, or the parameter counts are wrong.
Even a correctly calculated minimum has a limited meaning: it identifies the best-scoring model among the candidates evaluated, under the chosen criterion and data. It does not prove that model is true, that its assumptions hold, or that no important predictor or structure has been omitted.
Conditions for a Meaningful Criterion Comparison
For likelihood-based criteria, start by checking that the models are fitted to the same response values for the same observational units. They should also use the same response scale and likelihood convention. AIC and BIC have different penalties, so their numerical values are not interchangeable. Likewise, an AIC reported with constants retained cannot be directly compared with one from which constants have been removed.
For context, AIC is defined as \(-2\ell(\hat{\theta})+2K\), where \(\ell(\hat{\theta})\) is the maximized log-likelihood and \(K\) is the number of estimated parameters in that likelihood. BIC uses a different penalty, \(-2\ell(\hat{\theta})+K\log(n)\). AICc adds a small-sample correction to AIC. The criterion and convention must be consistent across the models being compared.
For a normal linear regression with an intercept, a common response, and independent normal errors with a common variance, one can use a reduced AIC for comparisons on the same \(n\) observations. Let \(p\) be the number of fitted regression coefficients, including the intercept. The full AIC counts the error variance as one additional estimated parameter; the reduced expression below drops terms common to all such models:
This reduction is safe for ranking only when \(n\), the response, and the likelihood setup are shared. It does not make scores from different samples comparable. Similarly, when using AICc or BIC, use the same full or reduced convention and count all parameters required by the formula consistently.
Error 1: Comparing Models Fitted to Different Observations
Different candidate models may have different missing-data patterns. If each model is fitted using every row available to it, one may be evaluated on 50 observations and another on only 42. Their maximized likelihoods then describe different sets of response values. Even if the response has the same name and units, the criteria no longer provide a clean comparison of how the models fit the same data.
Worked Example: A Smaller Score From a Different Sample
Suppose two hypothetical models predict the same response in the same units. Model A uses 50 observations, has \(\mathrm{SSE}=500\), and fits \(p=3\) regression coefficients. Model B uses 42 observations, has \(\mathrm{SSE}=420\), and fits \(p=4\) coefficients. Assume each satisfies the normal linear-model conditions on the observations actually used. Calculate each reduced AIC, then decide whether the scores can rank the models.
For Model A, \(n=50\), so:
For Model B, \(n=42\), so:
Model B's number is smaller, but it would be a mistake to conclude that Model B is preferred on this basis: it was fitted to a different set of observations. The calculations are each valid reduced scores for their respective fits, but the scores do not constitute a fair comparison on a common data set. A defensible comparison could fit both models to the same 42 observations, provided that this restriction is appropriate for the analysis, and recalculate both criteria there.
Error 2: Mixing Incompatible Score Conventions
An AIC value may be reported in full form or with terms common to the candidate models removed. Removing a common additive constant preserves the ranking only if that same constant is removed for every model. It is not valid to compare one model's reduced score with another model's full score.
In the normal regression setting described above, the full AIC is \(\ n[\log(2\pi)+1+\log(\mathrm{SSE}/n)]+2(p+1)\). Here the additional \(1\) in \(p+1\) counts the estimated error variance. The reduced AIC drops the common quantity \(n[\log(2\pi)+1]+2\). That quantity is the same across models only when they share the response, sample size, and likelihood setup.
Worked Example: Keep Both Scores on the Same Convention
Suppose two hypothetical normal regression models use the same 44 response observations. Model A has \(\mathrm{SSE}=440\) and \(p=4\) regression coefficients; Model B has \(\mathrm{SSE}=420\) and \(p=3\). Both models include an intercept, and the error-variance parameter is treated in the same way. Calculate their reduced AIC scores.
For Model A:
For Model B:
The difference is \(109.313744-105.266863\approx 4.046881\), or about 4.047 reduced-AIC units. Thus Model B has the smaller reduced AIC on this shared sample. The same ranking would result if both models were reported using full AIC, because the same constant would be added to each score.
By contrast, suppose Model A's reduced score were compared directly with Model B's full AIC. The full score adds \(44[\log(2\pi)+1]+2\approx126.8666\) to Model B's reduced value, giving about \(232.1335\). That number cannot be compared directly with Model A's reduced value of about \(109.3137\). The apparent difference would mostly reflect the mismatch in conventions, not a difference in model quality.
Error 3: Counting Parameters Incorrectly
A parameter count is not always the same as a count of predictor names. A categorical predictor with several levels usually requires multiple regression coefficients. For example, a four-level factor represented by indicator variables typically requires three coefficients when an intercept is included. The intercept is also a fitted coefficient. If the criterion's formula counts the error variance, that variance must be included too.
Make the counting rule explicit before comparing scores. In the reduced-AIC formula for normal regression above, \(p\) counts regression coefficients, including the intercept, while the common variance penalty has been omitted. In full AIC, \(K=p+1\) for this model because the error variance is also estimated. These are different bookkeeping conventions for different formulas; do not combine the count from one with the formula from the other.
Worked Example: A Factor With Several Levels
Consider two hypothetical models fitted to the same 60 observations, with a common continuous response. Model A has an intercept and one numeric predictor, so \(p=2\), and \(\mathrm{SSE}=600\). Model B has an intercept and a four-level categorical predictor represented by three indicators, so \(p=4\), and \(\mathrm{SSE}=570\). Assume the normal linear-model conditions hold and use reduced AIC for both.
For Model A:
For Model B, correctly counting the three indicator coefficients as well as the intercept:
With correct counting, Model A has the smaller score, by about \(143.0775-142.1551=0.9224\). Now suppose Model B is incorrectly treated as having only two coefficients because the analyst counts the factor as one predictor. Its mistaken score is \(60\log(9.5)+2(2)\approx139.0775\), which would incorrectly make Model B appear better. The parameter-counting error has changed the ranking.
Error 4: Treating the Minimum as Proof of Truth
A minimum criterion is a result of a comparison, not a certificate of truth. It identifies the lowest score among the models and conventions actually considered. If the candidate set omits an important predictor, uses an unsuitable functional form, or violates the assumptions underlying the fitted likelihood, the minimum cannot repair those problems. Adding or removing a candidate can also change which model has the smallest score.
Worked Example: The Best of a Limited Candidate Set
Suppose three hypothetical models are scored with the same information criterion, on the same response observations and with consistent parameter counting. Their scores are 90.0, 94.0, and 103.0. Model A therefore has the minimum, by 4.0 units relative to Model B and 13.0 units relative to Model C.
That arithmetic supports a limited conclusion: among these three candidates, Model A has the smallest value of the chosen criterion on these data. It does not establish that Model A is the true data-generating model or that its residual assumptions are satisfied. For example, if all three candidates force a straight-line relationship but the true relationship is curved, the lowest-scoring straight-line model may still be inadequate. Checking model assumptions and the suitability of the candidate models remains necessary.
A Short Justification: Why Common Constants Matter
Suppose the score for each candidate is \(C+S_j\), where \(C\) is exactly the same number for every model and \(S_j\) is the part that varies by model. For any two candidates \(a\) and \(b\):
The constant cancels, so removing it leaves the ranking unchanged. But if different constants are used for different models, say \(C_a\) and \(C_b\), the difference becomes \((C_a-C_b)+(S_a-S_b)\). The extra term need not be zero and can change the ranking. This is why reduced criteria require a genuinely common sample and convention; it is not a license to omit whatever terms are inconvenient.
A Practical Audit Before Reporting a Ranking
Before interpreting criterion values, check the comparison in a fixed order. This does not replace model diagnostics; it prevents avoidable comparison errors before the results are interpreted.
Verify that every candidate uses the same response values for the same observational units, and report the sample size.
Use the same criterion, likelihood, response scale, and full or reduced form throughout the comparison.
Count fitted coefficients, indicator terms, intercepts, and any other parameters required by the criterion's formula.
Identify the minimum only as the best-scoring candidate under the stated criterion, then assess whether the model assumptions and specification are reasonable.
Common Pitfalls
- “The model with the smaller number wins, even though it used fewer rows.” Scores from different response samples are not a fair direct ranking. Refit candidates to the same appropriate observations and recompute them.
- “The constants are unimportant, so any version of AIC can be compared.” Only common constants cancel. Use one convention for every model.
- “There are three predictors, so there are three parameters.” Predictor labels do not determine the parameter count. Count the coefficients actually fitted, and follow the criterion's definition of \(K\).
- “The minimum criterion proves the selected model is correct.” It establishes only a relative ranking within the evaluated candidate set. It does not test whether the candidate set or model assumptions are adequate.
Check Your Understanding
For each question, explain the reasoning rather than giving only a yes-or-no answer.
- Two models have AIC values calculated from different sets of response observations. What must be checked before their scores can be used for a direct comparison?
- Why does dropping a constant preserve a criterion ranking when the constant is the same for every model? What happens if it differs?
- A model has an intercept and a four-level factor coded with indicator variables. How many regression coefficients does it typically fit for that factor and intercept together?
- In the stated reduced-AIC convention for normal regression, why is the error-variance parameter not included in \(p\)?
- Three candidates were evaluated and one has the smallest criterion value. State one conclusion that is justified and two conclusions that are not justified by that minimum alone.