Why a Predictive Criterion Needs a Correction
A model fitted to observed losses or claim counts will usually describe those observations better than an independent sample from the same process. Some of that apparent improvement reflects genuine structure, but some comes from estimating the model parameters using the very data being assessed. A predictive criterion should account for this optimism: the gap between the model's fit to its training data and its expected fit to new data.
Akaike's information criterion, or AIC, makes an asymptotic correction for this optimism. Its penalty is not an arbitrary charge for complexity. Under regularity conditions, fitting \(k\) parameters improves the maximized training log likelihood by about \(k/2\), while it reduces expected test log likelihood by about \(k/2\). The training-to-test gap is therefore about \(k\) on the log-likelihood scale, or \(2k\) on the deviance scale.
For observations \(Y_1,\ldots,Y_n\) with log likelihood \(\ell(\theta)=\sum_{i=1}^n\log f_\theta(Y_i)\), the usual AIC is
where \(\hat\theta\) is the maximum likelihood estimate and \(k\) is the number of freely estimated parameters in the candidate model. Lower AIC indicates a smaller estimated predictive discrepancy among the candidates being compared. The criterion estimates predictive performance; it is not a test that a model is true.
Deriving the \(2k\) Correction
The derivation below uses a correctly specified, regular parametric model. Assume independent observations from \(f_{\theta_0}\), where the true parameter \(\theta_0\) is an interior point of a \(k\)-dimensional parameter space. Also assume the likelihood is sufficiently smooth, the parameter is identifiable, the Fisher information is finite and nonsingular, and the maximum likelihood estimator has its usual large-sample behavior. These conditions rule out, for example, many boundary and non-identifiable cases.
Let \(U=\sum_{i=1}^n s_i(\theta_0)\) be the total score at the true parameter and let \(I=E_0[s_i(\theta_0)s_i(\theta_0)^{T}]\) be the per-observation Fisher information matrix. To first order, \(\hat\theta-\theta_0\approx(nI)^{-1}U\), with covariance approximately \(I^{-1}/n\).
A second-order Taylor expansion of the training log likelihood around \(\theta_0\) gives a fitted gain of approximately \(\tfrac12 U^{T}(nI)^{-1}U\). Since \(E_0[UU^{T}]=nI\), the expected gain is \(\tfrac12\operatorname{tr}((nI)^{-1}nI)=k/2\).
Evaluate an independent test sample of size \(n\) at \(\hat\theta\). Conditional on the training estimate, its first-order score term has expectation zero. The second-order term has expectation approximately \(-\tfrac n2(\hat\theta-\theta_0)^T I(\hat\theta-\theta_0)\). Averaging over training samples gives \(-\tfrac n2\operatorname{tr}(I I^{-1}/n)=-k/2\).
The expected training log likelihood is about \(k/2\) above its value at the true parameter, while the expected test log likelihood is about \(k/2\) below that value. Thus the apparent training-to-test optimism is \(k/2-(-k/2)=k\) log-likelihood units. Multiplying by \(-2\) gives a deviance optimism of \(2k\).
The two contributions in the derivation have opposite signs. In-sample fitting raises the maximized log likelihood; estimation error lowers expected performance on independent data. Their difference, rather than either contribution alone, produces the correction.
Worked Example: Checking the Optimism in a Normal Location Model
Suppose \(X_1,\ldots,X_n\) are independent with \(X_i\sim N(\mu,\sigma^2)\), where \(\sigma^2\) is known and \(\mu\) is the one estimated parameter. Verify the expected training-to-test optimism in log-likelihood units.
Training fit. The maximum likelihood estimate is \(\bar X\). Relative to evaluating the training likelihood at the true \(\mu\), its log-likelihood gain is $$ \ell(\bar X)-\ell(\mu) =\frac{n(\bar X-\mu)^2}{2\sigma^2}. $$ Because \(\operatorname{Var}(\bar X)=\sigma^2/n\), the expected gain is \(n(\sigma^2/n)/(2\sigma^2)=1/2\).
Independent test fit. For a new sample \(X_1^*,\ldots,X_n^*\) from the same distribution, the expected test log-likelihood change from using \(\bar X\) instead of \(\mu\) is $$ E\!\left[-\frac{n(\bar X-\mu)^2}{2\sigma^2}\right] =-\frac{n}{2\sigma^2}\frac{\sigma^2}{n} =-\frac12. $$ The training-to-test optimism is therefore \(1/2-(-1/2)=1\) log-likelihood unit. On the deviance scale it is \(2(1)=2=2k\), since \(k=1\).
Check. A single estimated mean contributes the AIC penalty \(2\), not \(1\): the latter is the optimism on the log-likelihood scale before multiplying by \(-2\). This exact calculation for the normal location model illustrates the asymptotic derivation.
Applying the Correction to Candidate Models
AIC compares the fit and complexity of candidate models using the same observations and the same likelihood convention. The log-likelihood term rewards fit, while \(2k\) offsets the expected overstatement of that fit. The following examples show how to keep the signs and scales straight.
Worked Example: A Bernoulli Model With an Estimated Success Probability
A hypothetical portfolio records whether each of \(n=10\) reported small losses exceeds a specified threshold. Seven exceed it. Compare a fixed model with \(p=0.5\) to a Bernoulli model that estimates \(p\). Treat the observations as independent Bernoulli variables, and use natural logarithms.
Fixed model. Its parameter is not estimated from these data, so \(k_0=0\). The log likelihood is $$ \ell_0=7\log(0.5)+3\log(0.5)=10\log(0.5)=-6.9314718. $$ Thus \(-2\ell_0=13.8629436\), and \(\mathrm{AIC}_0=13.8629436+2(0)=13.8629436\).
Estimated model. The maximum likelihood estimate is \(\hat p=7/10=0.7\), and \(k_1=1\). Its log likelihood is $$ \ell_1=7\log(0.7)+3\log(0.3) =-2.4967246-3.6119184 =-6.1086430. $$ Therefore \(-2\ell_1=12.2172860\), and \(\mathrm{AIC}_1=12.2172860+2(1)=14.2172860\).
The signed change in deviance is \((-2\ell_1)-(-2\ell_0)=12.2172860-13.8629436=-1.6456576\). The estimated model improves the fit, but adding its parameter gives \(\Delta\mathrm{AIC}=-1.6456576+2=0.3543424\). Rounding only at the final step, to four decimal places, the AIC values are 13.8629 and 14.2173. The fixed model has the lower AIC by 0.3543.
Method and trap avoided. Compute each criterion as \(-2\ell+2k\), or calculate the signed difference as \(-2(\ell_1-\ell_0)+2(k_1-k_0)\). Reversing the model order changes the sign of the likelihood improvement; here the estimated model's deviance change is negative, not positive.
Worked Example: A Poisson Claim-Frequency Model
For a hypothetical group of four equal exposure units, the observed claim counts are \(0,1,2,3\). Compare a Poisson model with fixed rate \(\lambda=1\) to one with an estimated common rate. Assume independent counts with \(N_i\sim\operatorname{Poisson}(\lambda)\), and use natural logarithms.
The total count is \(0+1+2+3=6\), so the maximum likelihood estimate is \(\hat\lambda=6/4=1.5\). For counts \(n_i\), the log likelihood is $$ \ell(\lambda)=\sum_{i=1}^4\{n_i\log\lambda-\lambda-\log(n_i!)\}. $$ The factorial contribution is \(\log(0!)+\log(1!)+\log(2!)+\log(3!)=\log 12=2.4849066\).
Fixed rate. With \(k_0=0\), $$ \ell_0=6\log(1)-4(1)-\log(12)=-4-2.4849066=-6.4849066. $$ Hence \(\mathrm{AIC}_0=-2(-6.4849066)=12.9698132\).
Estimated rate. With \(k_1=1\), \(\log(1.5)=0.4054651\), so $$ \ell_1=6(0.4054651)-4(1.5)-2.4849066 =2.4327906-6-2.4849066 =-6.0521160. $$ Then \(-2\ell_1=12.1042320\), and \(\mathrm{AIC}_1=12.1042320+2=14.1042320\). The signed AIC difference is \(14.1042320-12.9698132=1.1344188\). To four decimal places, the AIC values are 12.9698 and 14.1042, so the fixed-rate model has the lower AIC for these data.
The estimated rate raises the log likelihood by \(0.4327906\), equivalent to a deviance decrease of \(2(0.4327906)=0.8655812\). That decrease is smaller than the one-parameter penalty of \(2\), leaving an AIC increase of \(2-0.8655812=1.1344188\). This check confirms the comparison's direction.
Assumptions, Interpretation, and Common Traps
The \(2k\) derivation is an asymptotic result, not a universal finite-sample identity. It relies on regular estimation and, in the derivation given here, correct specification. If a candidate is misspecified, the information governing score variability need not equal the information from the expected curvature. The simple \(2k\) correction then need not equal the optimism; a more general correction can involve a trace of those two information matrices.
AIC is useful because its target is expected predictive discrepancy, not a claim that the selected candidate is the true data-generating process. AIC values are meaningful as relative comparisons only when candidates describe the same observations with comparable likelihoods. Constants shared across candidates do not affect their ranking, but changing the data or likelihood scale between candidates can invalidate a direct comparison.
A frequent exam error is to state that each parameter adds \(2k\) log-likelihood units. It adds \(k\) expected log-likelihood units of optimism, which becomes \(2k\) after multiplying by \(-2\). Another error is to treat a better maximized likelihood as necessarily giving a better AIC: the fit improvement must be weighed against the penalty. In the Bernoulli example, the fitted model's deviance improved by 1.6457, less than its penalty of 2, so its AIC was higher.
Check Your Understanding
For each problem, compare AIC values or identify the correction using the assumptions stated. Lower AIC is preferred.
- Two models are fitted to the same data. Model A has \(\ell_A=-40.2\) and \(k_A=2\); model B has \(\ell_B=-39.5\) and \(k_B=3\). Calculate both AIC values and state which is lower. (Answer: \(\mathrm{AIC}_A=84.4\), \(\mathrm{AIC}_B=85.0\); model A is lower by 0.6.)
- Eight independent Bernoulli observations contain six successes. Compare fixed \(p=0.5\), with \(k=0\), against estimated \(p=0.75\), with \(k=1\). Use natural logs. (Answer: \(\ell_0=8\log(0.5)=-5.5452\), so \(\mathrm{AIC}_0=11.0904\); \(\ell_1=6\log(0.75)+2\log(0.25)=-4.4987\), so \(\mathrm{AIC}_1=10.9974\); the estimated model is lower by 0.0930.)
- In a normal model with known variance \(\sigma^2=4\) and one estimated mean, the fitted residual sum of squares is 20 and the fixed-mean sum of squares is 45. Find the estimated model's AIC change relative to the fixed model, given that the estimated model adds one parameter. (Answer: the deviance change is \((20-45)/4=-6.25\); adding the penalty of 2 gives \(\Delta\mathrm{AIC}=-4.25\), so the estimated model is lower.)
- Under the regular, correctly specified assumptions used in the derivation, the expected fitted training log likelihood is \(k/2\) above the true-parameter training value, and the expected independent test log likelihood is \(k/2\) below it. State the optimism on both scales. (Answer: \(k\) log-likelihood units and \(2k\) deviance units.)
- A candidate has \(\ell=-72.6\) and four freely estimated parameters. Calculate its AIC, and say what its value alone establishes. (Answer: \(-2(-72.6)+2(4)=153.2\); by itself, this AIC does not establish that the model is true or adequate, and it has no selection meaning without comparable candidate AIC values.)