Interpreting Differences In AIC

Why AIC Differences Need Interpretation

A list of AIC values ranks candidate models, but the gaps between those values also convey information. A difference of one unit and a difference of ten units do not indicate the same degree of relative support. Converting differences to relative likelihoods, and then to Akaike weights, gives a useful scale for comparing models in the stated candidate set.

These quantities must be interpreted carefully. An Akaike weight is not the probability that a model is true, nor is it automatically the probability that the model will be selected in a new sample. It is a normalized measure of relative support based on the AIC values of the candidate models being compared.

Definition: Suppose \(R\) candidate models are fitted to the same response observations using comparable likelihoods. For model \(i\), let \(\mathrm{AIC}_i=-2\ell_i+2k_i\), where \(\ell_i\) is its maximized log-likelihood and \(k_i\) is its number of estimated parameters. Define its AIC difference by \(\Delta_i=\mathrm{AIC}_i-\min_j\mathrm{AIC}_j\). The relative likelihood associated with model \(i\) is \(L_i=\exp(-\Delta_i/2)\). Its Akaike weight is \(w_i=L_i/\sum_{j=1}^R L_j\).

The minimum-AIC model has \(\Delta_i=0\), so its relative likelihood is \(1\). Every other model has a nonnegative difference and a relative likelihood between zero and one. The weights are nonnegative and sum to one, but their interpretation remains conditional on the candidates included in the comparison.

Why the Exponential Transformation Works

The factor of one-half and the negative sign follow directly from the AIC definition. Let model \(b\) have the lowest AIC. Subtracting its AIC from that of model \(i\) gives

\Delta_i=\mathrm{AIC}_i-\mathrm{AIC}_b =-2(\ell_i-\ell_b)+2(k_i-k_b).

Multiplying by \(-1/2\) and exponentiating yields

\exp(-\Delta_i/2) =\exp\big((\ell_i-\ell_b)-(k_i-k_b)\big) =\frac{\exp(\ell_i)}{\exp(\ell_b)}\exp(-(k_i-k_b)).

Thus, the transformation compares models using both their maximized likelihoods and their parameter-count penalties. It is not generally the ratio of the models’ maximized likelihoods alone: that ratio would omit the penalty term \(\exp(-(k_i-k_b))\). The AIC-based relative likelihood is a conventional measure of relative support, not a posterior probability derived from Bayes’ theorem.

Worked Example: Turning AIC Gaps Into Weights

A pricing analyst compares three hypothetical regression models fitted to the same 160 policy observations. Their AIC values are 214.6, 216.6, and 220.6. Calculate the differences, relative likelihoods, and Akaike weights. State what the results do and do not imply.

Method and check. First subtract the smallest AIC from each value. Then apply \(\exp(-\Delta_i/2)\), and divide each relative likelihood by their sum. The candidates use the same observations, as required for this comparison.

The minimum AIC is 214.6. The differences are

\Delta_1=214.6-214.6=0,\qquad \Delta_2=216.6-214.6=2,\qquad \Delta_3=220.6-214.6=6.

The relative likelihoods are

L_1=\exp(0)=1,\qquad L_2=\exp(-2/2)=\exp(-1)=0.3679,\qquad L_3=\exp(-6/2)=\exp(-3)=0.0498.

Their sum is \(1+0.3679+0.0498=1.4177\). Dividing each value by this sum gives

w_1=1/1.4177=0.7054,\qquad w_2=0.3679/1.4177=0.2595,\qquad w_3=0.0498/1.4177=0.0351.

The weights sum to \(0.7054+0.2595+0.0351=1.0000\), to four decimals. Within these three candidates, the first model has the greatest AIC-based support. Its weight is about 2.7 times the second model’s weight, since \(0.7054/0.2595\approx2.7\). This does not mean there is a 70.54% probability that the first model is true; the weights are normalized only over the candidates in this comparison.

Common mistake avoided: Treating the AIC gap of 2 as a relative likelihood of \(2\), or using \(\exp(2/2)\), reverses the direction. A higher AIC means less relative support, so the correct calculation is \(\exp(-2/2)=0.3679\).

Calculating the Weights From Log-Likelihoods

If a problem supplies maximized log-likelihoods and parameter counts rather than AIC values, calculate each AIC first. Use the same AIC convention for every candidate. In a Gaussian regression comparison, for example, the count \(k_i\) must include every estimated parameter under the convention used for the likelihood, including the error-variance parameter when it is estimated. Do not compare a log-likelihood from one set of observations with an AIC based on another.

Worked Example: From Model Fits to Normalized Weights

A hypothetical claims team fits three models to the same 80 claim records. The supplied maximized log-likelihoods and parameter counts are: model P, \(\ell_P=-48.2\) and \(k_P=4\); model Q, \(\ell_Q=-46.5\) and \(k_Q=6\); model R, \(\ell_R=-49.0\) and \(k_R=3\). Calculate the AIC values, differences, relative likelihoods, and weights.

Method. Apply \(\mathrm{AIC}=-2\ell+2k\) to each model, then calculate differences from the minimum. This makes the likelihood improvement and the complexity penalty explicit.

For P,

\mathrm{AIC}_P=-2(-48.2)+2(4)=96.4+8=104.4.

For Q,

\mathrm{AIC}_Q=-2(-46.5)+2(6)=93.0+12=105.0.

For R,

\mathrm{AIC}_R=-2(-49.0)+2(3)=98.0+6=104.0.

The lowest AIC is 104.0, from R. Therefore, \(\Delta_R=0\), \(\Delta_P=104.4-104.0=0.4\), and \(\Delta_Q=105.0-104.0=1.0\). The relative likelihoods, rounded to four decimals, are

L_R=1,\qquad L_P=\exp(-0.4/2)=\exp(-0.2)=0.8187,\qquad L_Q=\exp(-1.0/2)=\exp(-0.5)=0.6065.

Their sum is \(1+0.8187+0.6065=2.4252\). Thus,

w_R=1/2.4252=0.4123,\qquad w_P=0.8187/2.4252=0.3376,\qquad w_Q=0.6065/2.4252=0.2501.

The rounded weights add to 1.0000. Although R has the smallest AIC, the three weights are not widely separated: P and Q retain substantial relative support in this candidate set. A common error would be to choose Q because its log-likelihood is greatest. AIC accounts for both fit and parameter count; Q’s improved log-likelihood does not offset its larger penalty enough to produce the smallest AIC.

Weights Depend on the Candidate Set

The relative likelihood \(L_i\) is calculated from the difference between model \(i\)’s AIC and the minimum AIC. If a new candidate is added but none of the original AIC values changes, the original pairwise AIC differences and relative likelihoods stay the same. The normalized weights can nevertheless change because their denominator now includes the new model.

This distinction matters when reporting results. State which candidates were compared, and avoid presenting a weight as an absolute measure of evidence. A weight of 0.50 in one candidate set need not remain 0.50 in a different set.

Worked Example: How Adding a Candidate Changes Weights

A hypothetical risk analyst has compared three models with AIC values 301, 303, and 307. The analyst then adds a fourth candidate with AIC 302. Find the original weights and the revised weights, and identify which quantities for the original three candidates change.

Method. Calculate the original differences and relative likelihoods, normalize them, then repeat for all four candidates. The AIC minimum remains 301, so the original differences do not change.

For the first three models, the differences are \(0, 2,\) and \(6\), so their relative likelihoods are \(1, 0.3679,\) and \(0.0498\). The sum is \(1.4177\). The original weights are therefore \(0.7054, 0.2595,\) and \(0.0351\), to four decimals.

For the added model, the difference is \(302-301=1\), giving relative likelihood \(\exp(-1/2)=0.6065\). The revised denominator is

$$ 1+0.3679+0.0498+0.6065=2.0242. $$

The four revised weights are

1/2.0242=0.4940,\quad 0.3679/2.0242=0.1817,\quad 0.0498/2.0242=0.0246,\quad 0.6065/2.0242=0.2996.

They sum to 0.9999 because of rounding; before rounding, the weights sum to 1. The original models’ AIC differences and relative likelihoods are unchanged, but their normalized weights decrease because the denominator has increased. The new candidate’s weight describes its share of the normalized AIC-based support among these four candidates, not a probability that it is the true model.

Common mistake avoided: Keeping the old normalized weights after adding a candidate is incorrect. Recalculate the denominator whenever the candidate set changes.

Interpreting Results Responsibly

AIC differences and weights are useful summaries, but they answer a limited question: how do the candidate models compare under AIC, given the data and the specified candidate set? They do not show that every potentially useful model was considered. Nor do they account for uncertainty about model form in the same way as a Bayesian posterior model probability.

Before interpreting the calculations, check that the models use the same response observations and a comparable likelihood basis. In particular, likelihood-based criteria can be misleading when models use different samples or incompatible response scales. The earlier material on common-sample comparisons and response transformations addresses those comparability issues; a weight calculation cannot repair a comparison that was not valid to begin with.

A compact interpretation should name the minimum-AIC model, describe the differences or relative weights, and identify the candidate set. For example: “Among these three models, model P has the smallest AIC; model Q’s relative likelihood is 0.37 compared with P, and the normalized weights are conditional on these three candidates.” This is more accurate than saying, “P has a 71% chance of being correct.”

Key takeaway: Compute \(\Delta_i=\mathrm{AIC}_i-\min_j\mathrm{AIC}_j\), then \(L_i=\exp(-\Delta_i/2)\), and normalize using \(w_i=L_i/\sum_jL_j\). Relative likelihoods compare AIC support to the best candidate; Akaike weights allocate that support across the stated candidate set. Neither quantity is, by itself, a posterior probability that a model is true.

Check Your Understanding

For each problem, calculate the requested quantities and interpret them in the stated candidate set.

A hypothetical severity analysis produces AIC values 88, 91, and 94 for models A, B, and C. Find their AIC differences and relative likelihoods, to four decimals. (Answer: Differences \(0, 3, 6\); relative likelihoods \(1.0000, 0.2231, 0.0498\).)
Two models fitted to the same portfolio have AIC values 412.8 and 413.8. Find their differences, relative likelihoods, and normalized Akaike weights, to four decimals. (Answer: Differences \(0, 1\); relative likelihoods \(1.0000, 0.6065\); weights \(0.6225, 0.3775\).)
Three hypothetical models have AIC values 150, 152, and 152. Calculate all three Akaike weights to four decimals. (Answer: Relative likelihoods \(1.0000, 0.3679, 0.3679\); weights \(0.5761, 0.2119, 0.2119\).)
A candidate model has an AIC difference of 4 from the minimum. Calculate its relative likelihood and explain what the result means. (Answer: \(\exp(-4/2)=0.1353\); its AIC-based relative likelihood is 0.1353 compared with the minimum-AIC model, not a 13.53% probability that it is true.)
A new model is added to a comparison without changing any original model’s AIC, and the new model has a higher AIC than the original minimum. Which quantities for the original models must stay fixed, and which may change? (Answer: Their AIC differences and relative likelihoods stay fixed; their normalized Akaike weights may decrease because the sum used to normalize them increases.)

What You'll Learn