When the Scores Are Nearly Tied
Information criteria give a useful way to compare regression models, but their rankings can look more decisive than the evidence warrants. If the best model has a score only slightly lower than the next-best model, the ranking still identifies the numerical minimum; it does not show that the first model is clearly better. Such a near-tie is a reason to acknowledge uncertainty about which candidate model gives the most useful account of the data.
This matters because choosing a model can affect which predictors appear in the analysis, the estimated regression coefficients, and the predictions or scientific conclusions drawn from the fit. A careful comparison distinguishes the model with the smallest score from a model that is clearly separated from its alternatives. The first claim follows directly from the computed scores. The second requires a meaningful gap, and even then it is a comparison within the candidate models considered—not proof that one model is uniquely correct.
What a Small Difference Means
For a chosen criterion, define a model’s difference from the best-scoring candidate as its criterion value minus the minimum value. For AIC, if candidate model \(i\) has score \(\mathrm{AIC}_i\), then its difference is
The smallest difference is zero; all other differences are nonnegative. The calculation makes the ranking easy to read, but a difference is not a test statistic, a p-value, or a measure of the probability that a model is correct. It summarizes how far a candidate’s score is from the minimum on that criterion’s scale.
As a rough convention for AIC comparisons, differences of about 0 to 2 often indicate substantial support for more than one candidate; differences around 4 to 7 indicate appreciably less support; and differences greater than 10 indicate very little support relative to the best candidate. These are guidelines, not universal cutoffs or guarantees. The strength of evidence also depends on the candidates being compared and the purpose of the analysis. A difference of 1.5 does not mean “1.5 units more likely,” and a difference of 2 is not a boundary at which a model suddenly becomes unacceptable.
A useful justification comes from the relative likelihood associated with an AIC difference. For candidate \(i\), its relative likelihood compared with the best-scoring model is \(\exp(-\Delta_i/2)\). This follows because AIC is \(-2\) times a maximized log likelihood plus a penalty: subtracting the best AIC value and dividing by \(-2\) gives the log of the relative support on this scale. Exponentiating returns the relative likelihood. For a small difference, this ratio is near 1, so the scores do not sharply distinguish the two candidates.
For example, a difference of 0.7 gives \(\exp(-0.7/2)\approx 0.705\). On this relative-likelihood scale, that candidate retains about 70.5% of the best model’s support, rather than being negligible. The comparison says nothing by itself about whether either model is adequate in an absolute sense.
Worked Example: Two Models With a Small AIC Difference
Suppose two hypothetical Gaussian least-squares regression models are fitted to the same response observations, with the same response scale and the same likelihood basis. Assume their AIC calculations include the appropriate parameter counts and have been checked. Model A has AIC \(120.0\), and model B has AIC \(121.4\). Compare them.
The minimum AIC is \(120.0\), so the differences are
Model A is the numerical winner. But a difference of 1.4 is small by the usual AIC guidelines. The relative likelihood for B compared with A is
Thus model B has about half the relative support of model A on this scale. That is not a negligible comparison, so it would be misleading to report only that A “wins” and imply that the choice is unambiguous. A reasonable account names A as the minimum-AIC model while noting that B is also supported by the comparison. Neither score establishes that a model is true or that its assumptions are satisfied.
Akaike Weights and the Candidate Set
Akaike weights put the relative likelihoods for a set of candidates onto a common scale that sums to 1. They are useful for describing how support is distributed across the specified set, especially when several models have small differences. They do not change the scores or eliminate uncertainty.
Each weight is between 0 and 1, and the weights sum to 1 over the candidate set. A weight describes a model’s relative share of this normalized support within that set. It is not the probability that the model is true, nor does it include models that were never considered. Changing the candidate set can change every normalized weight, even if the AIC values of models retained in both sets stay the same.
Worked Example: Several Candidates With Unequal but Shared Support
Consider four hypothetical regression models fitted to the same response data, with comparable likelihoods and correctly calculated AIC values: \(120.0\), \(120.7\), \(122.6\), and \(130.8\). Assume the model fits and the use of AIC are appropriate for this comparison. Find the differences and approximate Akaike weights, then interpret the result.
Subtract the minimum score, \(120.0\), from each value. The differences are \(0.0\), \(0.7\), \(2.6\), and \(10.8\). Their relative likelihoods are
The unrounded relative likelihoods sum to approximately \(1.9817\). Dividing each by that sum gives weights of approximately \(0.5046\), \(0.3556\), \(0.1375\), and \(0.0023\). The first two candidates therefore have a combined weight of
The minimum-AIC model has the largest individual weight, but the second model also has substantial support; together, the first two account for about 86.02% of the normalized support in this four-model set. The third candidate has less support, and the fourth has very little relative support compared with the best. These weights do not imply that the first model has a 50.46% probability of being true. They describe the relative distribution of support among these four candidates under the stated comparison.
Reporting Several Plausible Models
When scores are near-tied, reporting a small set of plausible models can communicate the evidence more faithfully than presenting one as uniquely correct. A useful report identifies the criterion used, gives the scores or differences for the relevant candidates, states which model has the smallest score, and explains that the close alternatives remain plausible. If weights are included, state the candidate set to which they apply and avoid translating them into probabilities of truth.
The purpose is not to list every fitted model regardless of support. It is to make clear when the criterion has not strongly separated a few candidates. The choice of which candidates to examine should also be scientifically defensible: a low score does not rescue a model with an implausible structure or serious lack of fit. In particular, an information-criterion comparison does not replace checking whether the regression model is suitable for the data and question.
Worked Example: Describing a Near-Tie Without Overclaiming
In a hypothetical analysis of a continuous response measured in milligrams, three candidate models use the same observations and response scale. All are fitted using the same likelihood basis, and their assumptions and calculations have been checked. Their AIC values are \(84.2\), \(85.0\), and \(88.9\). The first two models estimate the association with a predictor differently: their estimated slopes are \(1.8\) and \(2.2\) milligrams per unit of that predictor. Write a cautious comparison.
The differences are
The first model has the smallest AIC. The difference of \(0.8\) between the first and second models is small, so AIC provides limited separation between them. Their estimated slopes differ by
The third model is less supported than the first two under this comparison, though its difference alone does not prove it unsuitable. A measured report would say that the first model had the minimum AIC, that the second model was also close by AIC, and that their estimated slopes were \(1.8\) and \(2.2\) milligrams per predictor unit. The similarity of the scores does not make the slope estimates identical; showing both results helps readers see that the estimated association depends somewhat on the model choice. This comparison does not establish a causal effect.
Why Near-Ties Matter
A single selected model can make later results appear more certain than the comparison warrants. If plausible alternatives produce materially different coefficient estimates, predicted values, or interpretations, then that variation is relevant to the analysis. Reporting more than one supported model makes the dependence on modeling choices visible. It does not, by itself, combine the models into a single estimate or quantify all sources of uncertainty; it is a transparent account of uncertainty about model choice.
Near-ties also have a practical meaning: the observed data and chosen criterion have not clearly distinguished the candidates. A different sample could rank them differently, particularly when the candidates describe similar patterns. Treat the ranking as evidence conditional on the data, criterion, and candidate set—not as a permanent ordering of models in every possible sample.
Common Pitfall
Wrong reasoning: “Model A has the lowest AIC, so it is the correct model; the other models can be ignored.” The lowest score does identify the minimum among the candidates, but a small gap does not justify calling that model uniquely correct. Conversely, saying that near-tied models are “equally likely to be true” also overstates what the comparison shows.
Correction: State which candidate has the lowest score, then examine the size of the differences and, when useful, the distribution of weights. If several candidates have small differences, report them as plausible alternatives within the specified set. Explain how their substantive conclusions compare, and do not interpret a weight as a probability that a model is true.
Check Your Understanding
Use the definitions and interpretations above to answer the following questions.
- Two models have AIC values of \(210.0\) and \(211.2\). What are their AIC differences, and why would it be misleading to describe the second model as having no support?
- In words, what does an Akaike weight describe, and what does it not describe?
- Why can adding or removing candidate models change the Akaike weights even when the retained models’ AIC differences from the minimum are unchanged?
- A researcher reports only the minimum-AIC model even though another candidate has a difference of \(0.5\). What additional information would help readers assess model uncertainty?
- Why does a near-tie in AIC not show that either candidate fits the data adequately?