Deriving AIC's Two-Parameter Penalty

Why Fitted Training Likelihood Is Optimistic

A model’s parameters are estimated using the observed data. As a result, the fitted model has been tuned to features of that particular sample, including chance fluctuations. Its likelihood on the same data will therefore tend to look better than its likelihood on a fresh sample of the same size. This is training-fit optimism: the expected gap between the fitted training fit and independent test fit.

AIC corrects for this advantage. The key result is that, under regular likelihood assumptions, estimating $k$ parameters creates an expected training-versus-test log-likelihood gap of approximately $k$. Because AIC uses negative twice the log likelihood, the corresponding correction is approximately $2k$. This is a large-sample result, not an exact correction for every model and data set.

Definition: Let $\ell_D(\theta)$ be the log likelihood for a training data set $D$, evaluated at parameter value $\theta$, and let $\hat{\theta}$ be the estimate obtained from $D$. Let $D'$ be an independent data set of the same size from the same population. The training-fit optimism is $$ E\!\left[\ell_D(\hat{\theta})-\ell_{D'}(\hat{\theta})\right]. $$ The expectation is over both the training data and the independent test data. Positive optimism means that fitted training log likelihood is better, on average, than test log likelihood.

The comparison uses the same fitted estimate $\hat{\theta}$ in both likelihoods: it is fitted on $D$, then evaluated on $D'$. The test data do not get used to refit the parameters. If each sample were used to fit its own model, the comparison would combine training advantage with differences between two estimates.

Assumptions Behind the Approximation

The derivation below concerns a regular, correctly specified likelihood model with a fixed, finite number $k$ of free parameters. The observations are independent and identically distributed from the model at a true parameter value $\theta_0$. The true value is an interior point of the parameter space, the parameter is identifiable, the log likelihood is sufficiently smooth, and the Fisher information is finite and nonsingular. The maximum likelihood estimate is assumed to have the usual large-sample behavior under these conditions.

These assumptions allow the log likelihood near $\theta_0$ to be approximated by a quadratic function. They also make the estimate’s sampling variation predictable. The result is approximate because it relies on this large-sample, local description; it does not say that every fitted model has exactly $k$ log-likelihood units of optimism in a particular sample.

Notation for the derivation: Write $\ell_D(\theta)$ for the training log likelihood, $U_D(\theta_0)$ for its score (the vector of first derivatives) at the true parameter, and $I$ for the Fisher information per observation at $\theta_0$. Thus the expected negative Hessian for $n$ observations is $nI$. The parameter vector has $k$ components.

Deriving the Two-Per-Parameter Correction

Let $\delta=\hat{\theta}-\theta_0$. For a large sample, a Taylor expansion of the training log likelihood about $\theta_0$ gives a linear score term and a quadratic curvature term. The maximum likelihood estimate is approximately the value that balances those terms. Under the stated assumptions,

\delta \approx (nI)^{-1}U_D(\theta_0), \qquad \operatorname{Var}(\delta)\approx \frac{1}{n}I^{-1}.

Substituting the first approximation into the quadratic expansion gives the improvement in training log likelihood from estimating $\theta$, compared with evaluating at the true value:

\ell_D(\hat{\theta})-\ell_D(\theta_0) \approx U_D(\theta_0)^{\mathsf T}\delta -\frac{1}{2}\delta^{\mathsf T}(nI)\delta \approx \frac{1}{2}U_D(\theta_0)^{\mathsf T}(nI)^{-1}U_D(\theta_0).

The score has mean zero and covariance $nI$. Therefore the expected quadratic form in the final expression is approximately $k$: a vector with $k$ components contributes one unit on average for each component after standardization by its covariance. The expected training improvement is consequently about $k/2$ log-likelihood units.

Now evaluate the same estimate on the independent test data. Conditional on $\hat{\theta}$, the expected test score at $\theta_0$ is zero, and the expected curvature is approximately $-nI$. The expected test log likelihood therefore changes from its value at $\theta_0$ by approximately

-\frac{1}{2}E\!\left[\delta^{\mathsf T}(nI)\delta\right] \approx -\frac{1}{2}\operatorname{tr}\!\left(nI\operatorname{Var}(\delta)\right) \approx -\frac{k}{2}.

The trace is the sum of the diagonal elements of a matrix; here $nI\operatorname{Var}(\delta)\approx II^{-1}$, whose trace is $k$. Thus, relative to the true parameter, fitting raises training log likelihood by about $k/2$, while estimating from training data lowers expected test log likelihood by about $k/2$. Since the expected training and test log likelihoods at the fixed true parameter are equal, their fitted gap is about $k/2+k/2=k$.

Result: Under the regular, correctly specified large-sample assumptions above, the expected training-fit optimism is approximately $k$ log-likelihood units. Multiplying by negative two converts this to approximately $2k$ units on the deviance scale. This motivates $$ \mathrm{AIC}=-2\ell_D(\hat{\theta})+2k, $$ where $\ell_D(\hat{\theta})$ is the maximized log likelihood and $k$ counts the free parameters in the fitted likelihood.

AIC adds the correction to the observed training criterion, so a smaller AIC is preferred among the candidate models being compared. The penalty is not an arbitrary fee for using a complicated formula: it approximates the expected optimism caused by fitting the parameters. AIC is commonly used as an estimate of expected predictive information loss, with details of interpretation depending on the comparison and model assumptions.

Worked Example: Turning Parameter Count Into Expected Optimism

Suppose a hypothetical regular model has $k=2$ parameters and is fitted to $n=100$ independent observations. Assume the stated likelihood conditions hold. The parameter estimate has approximate covariance $I^{-1}/100$, while the total expected curvature is $100I$.

The expected quadratic form governing the training improvement is approximately $$ \operatorname{tr}\!\left(100I\frac{I^{-1}}{100}\right) =\operatorname{tr}(I I^{-1}) =\operatorname{tr}(I_2)=2. $$ The factor of one half in the likelihood expansion gives an expected training improvement of $2/2=1$ log-likelihood unit.

For independent test data, the expected change from the true-parameter log likelihood is approximately $-2/2=-1$ log-likelihood unit. The expected fitted training-versus-test gap is therefore $1-(-1)=2$ log-likelihood units, equal to $k$. On the negative-two-log-likelihood scale, the correction is $2(2)=4$. This is an expectation under the assumptions, not a claim that this particular sample’s measured gap must equal 2.

Using the Penalty to Compare Models

The formula makes the fit-complexity tradeoff explicit. A model with more parameters can achieve a higher maximized likelihood, but its AIC also includes a larger penalty. An added parameter is worthwhile for AIC only when the improvement in fit is large enough to offset that additional penalty.

Worked Example: Comparing Two Hypothetical Likelihood Models

Suppose two candidate models are fitted to the same hypothetical observations using the same response likelihood. Assume both are regular and that the information-criterion comparison is appropriate. Model A has $k_A=3$ parameters and maximized log likelihood $\ell_A=-80$. Model B has $k_B=5$ parameters and maximized log likelihood $\ell_B=-77$.

For Model A, $$ \mathrm{AIC}_A=-2(-80)+2(3)=160+6=166. $$ For Model B, $$ \mathrm{AIC}_B=-2(-77)+2(5)=154+10=164. $$ The more complex model improves the maximized log likelihood by $-77-(-80)=3$ units. On the negative-two scale, that fit improvement is $2(3)=6$ units. Its two additional parameters add $2(2)=4$ penalty units, so its AIC is lower by $6-4=2$. Under AIC, Model B is preferred in this comparison. That does not establish that it is the true model or that its predictions will be better in every future sample.

The same arithmetic can be used with a simple likelihood to see how the penalty is applied. The following example assumes a Bernoulli model: independent binary observations share one probability $p$, which is estimated from the data. This is a one-parameter regular model when the true probability is strictly between zero and one.

Worked Example: A One-Parameter Bernoulli Likelihood

In a hypothetical set of $n=10$ independent binary observations, 7 are successes and 3 are failures. Assume the observations follow the stated Bernoulli model with a common probability $p$. The log likelihood is $$ \ell(p)=7\ln(p)+3\ln(1-p). $$ The maximum likelihood estimate is $\hat p=7/10=0.7$, an interior estimate. Substituting it gives $$ \ell(\hat p)=7\ln(0.7)+3\ln(0.3) \approx 7(-0.3567)+3(-1.2040) \approx -6.109. $$

There is one estimated parameter, so $k=1$. The fitted negative-two log likelihood is approximately $-2(-6.109)=12.22$, and the AIC is $$ \mathrm{AIC}=12.22+2(1)=14.22. $$ The 2-unit addition is the approximate optimism correction for one estimated parameter. It is not an observed test error and does not mean that this particular fit is exactly 2 units too optimistic.

When to Be Careful

The derivation explains both the strength and the limits of the $2k$ penalty. It is a useful general approximation when the model and likelihood satisfy the regularity conditions, the sample is large enough for the quadratic approximation to be reasonable, and the models are compared on a compatible basis. It is not a universal finite-sample identity.

If the likelihood is misspecified, the curvature of the likelihood and the variability of its score need not match. In that case, the optimism correction need not equal $k$ on the log-likelihood scale; a more general correction can depend on both quantities. AIC’s ordinary $2k$ form should therefore not be justified by this derivation when its assumptions are substantially violated. Also, the derivation is about expected fit optimism, not a guarantee that the selected model is scientifically correct.

Worked Example: A Common Penalty Mistake

A hypothetical comparison finds that adding two parameters improves maximized log likelihood by 1.5 units. A student reasons: “The log likelihood improved, so the larger model must have the smaller AIC.”

That conclusion overlooks the penalty. The improvement on the negative-two-log-likelihood scale is $2(1.5)=3$. The two added parameters increase the penalty by $2(2)=4$. The net AIC change is $$ \Delta\mathrm{AIC}=-3+4=1. $$ The larger model’s AIC is 1 unit higher, so the smaller model has the lower AIC. The correction is applied on the same negative-two scale as the likelihood term; comparing a raw log-likelihood gain directly with a $2k$ penalty gives the wrong threshold.

Key takeaway: Estimating $k$ regular model parameters creates approximately $k$ log-likelihood units of expected training advantage over an independent test sample of equal size. On the negative-two-log-likelihood scale, that becomes the AIC penalty $2k$. Use the result as a large-sample approximation under its assumptions, not as an exact statement about one fitted data set.

Check Your Understanding

Answer each question by referring to the scale and assumptions involved.

In the regular large-sample derivation, what is the approximate training-versus-test gap in log-likelihood units for a model with $k=4$ parameters?
Why does an expected optimism of $k$ log-likelihood units become a $2k$ penalty in AIC?
A larger model adds three parameters and improves maximized log likelihood by 2 units. What is the resulting change in AIC?
Why must the independent test data be evaluated using the estimate fitted on the training data for the optimism definition?
Name two assumptions that support the quadratic likelihood approximation used in the derivation.