Deriving AIC From The Likelihood

Why AIC Penalizes Model Complexity

A model with more parameters can usually fit the observed data at least as well as a simpler model. Comparing maximized likelihoods alone would therefore reward complexity without accounting for the tendency to fit random noise. The Akaike information criterion (AIC) addresses this by estimating how much better a fitted model appears on its training data than it is expected to perform on new data from the same process.

AIC is based on the maximized log likelihood and a correction for this optimism. The correction is expressed in deviance units, which are minus twice the log-likelihood scale. The derivation explains why the correction is twice the number of free parameters, rather than simply the parameter count.

Definition: For a likelihood $L(\theta)$ with $k$ free parameters and maximum-likelihood estimate $\hat{\theta}$, the Akaike information criterion is $$ \mathrm{AIC}=-2\ell(\hat{\theta})+2k, \qquad \ell(\theta)=\log L(\theta). $$ A smaller AIC indicates a better estimated predictive fit among the candidate models being compared. AIC is not a probability that a model is true, and its numerical value has no standalone interpretation as a measure of model quality.

The usual $2k$ correction follows from a large-sample approximation for a regular, identifiable likelihood model with a finite-dimensional parameter in the interior of its parameter space. In the basic derivation, the model is correctly specified: the data are generated by a parameter value in the model. For independent observations, the log likelihood is a sum of individual log likelihoods. Under these conditions, fitting the parameters produces an average optimism of about $k$ log-likelihood units on a fresh sample of the same size. Multiplying by minus two gives the AIC penalty $2k$.

Why the Penalty Is Twice the Parameter Count

The key is to compare the fitted log likelihood on the data used for estimation with the expected log likelihood on an independent sample of the same size. Write the true parameter as $\theta_0$, the training-sample score as $U$, and the per-observation Fisher information at $\theta_0$ as $I$. For a regular model, a first-order approximation to the maximum-likelihood estimate is

\hat{\theta}-\theta_0 \approx \frac{1}{n}I^{-1}U, \qquad \operatorname{Var}(U)\approx nI.

A second-order Taylor expansion of the training log likelihood around $\theta_0$ gives an approximate gain from estimating rather than fixing the parameter at its true value. The score has mean zero, and the expected quadratic score term is $k$:

\begin{aligned} \ell_{\mathrm{train}}(\hat{\theta})-\ell_{\mathrm{train}}(\theta_0) &\approx U^\mathsf{T}(\hat{\theta}-\theta_0) -\frac{n}{2}(\hat{\theta}-\theta_0)^\mathsf{T}I(\hat{\theta}-\theta_0)\\ &\approx \frac{1}{2n}U^\mathsf{T}I^{-1}U,\\ E\!\left[\frac{1}{n}U^\mathsf{T}I^{-1}U\right] &\approx \frac{1}{n}\operatorname{tr}\!\left(I^{-1}\operatorname{Var}(U)\right) =\operatorname{tr}(I^{-1}I)=k. \end{aligned}

Thus, the expected training log-likelihood gain is about $k/2$. On a new independent sample, the parameter estimate is still displaced from $\theta_0$, but the new sample’s score has mean zero conditional on the training estimate. Its expected log likelihood is therefore lower than the value at $\theta_0$, by about $k/2$. Combining the two comparisons, training maximized log likelihood overstates new-sample log likelihood by about $k$:

\underbrace{\frac{k}{2}}_{\text{training gain from fitting}} - \underbrace{\left(-\frac{k}{2}\right)}_{\text{expected new-sample change}} =k.

AIC corrects this optimism on the minus-two-log-likelihood scale. Since multiplying a log-likelihood difference of $k$ by $-2$ changes the criterion by $2k$, the resulting expression is $-2\ell(\hat{\theta})+2k$. This is an asymptotic justification, not an exact identity for every sample or every model. If the likelihood is misspecified or nonregular, the simple $2k$ correction need not accurately represent predictive optimism.

How the Gaussian Regression Likelihood Produces the Fit Term

Consider a Gaussian linear regression with fixed predictor values and independent errors $Y_i=x_i^\mathsf{T}\beta+\varepsilon_i$, where $\varepsilon_i\sim N(0,\sigma^2)$, for $i=1,\ldots,n$. The variance is positive and common to all observations. Conditional on the predictors, the likelihood for the response data is

L(\beta,\sigma^2) =(2\pi\sigma^2)^{-n/2} \exp\!\left(-\frac{\mathrm{RSS}(\beta)}{2\sigma^2}\right), \qquad \mathrm{RSS}(\beta)=\sum_{i=1}^n (y_i-x_i^\mathsf{T}\beta)^2.

For any fixed $\sigma^2>0$, maximizing the likelihood over $\beta$ is equivalent to minimizing $\mathrm{RSS}(\beta)$, which is achieved by a least-squares fit. Denote the resulting residual sum of squares by $\mathrm{RSS}$. If $\mathrm{RSS}>0$, maximizing over the positive variance gives $\hat{\sigma}^2=\mathrm{RSS}/n$. Substitution into the likelihood yields the Gaussian fit term:

\begin{aligned} -2\ell(\hat{\beta},\hat{\sigma}^2) &=n\log(2\pi\hat{\sigma}^2)+\frac{\mathrm{RSS}}{\hat{\sigma}^2}\\ &=n\log\!\left(2\pi\frac{\mathrm{RSS}}{n}\right)+n\\ &=n\left[\log(2\pi)+1+\log\!\left(\frac{\mathrm{RSS}}{n}\right)\right]. \end{aligned}

The appearance of $n$ in the variance estimate is a maximum-likelihood result: it comes from maximizing the likelihood, not from an unbiased variance estimator. The fitted likelihood rewards smaller residual variation through the logarithm of $\mathrm{RSS}/n$. To obtain AIC, add $2k$, where $k$ is the number of free parameters in the likelihood. If the variance is estimated, it is one of those parameters.

The derivation requires $\mathrm{RSS}>0$. If $\mathrm{RSS}=0$, the likelihood is $(2\pi\sigma^2)^{-n/2}$, which grows without bound as $\sigma^2$ decreases to zero. There is then no finite maximum-likelihood estimate for a strictly positive variance, so the displayed maximized-likelihood expression and ordinary AIC calculation do not apply.

Worked Example: Deriving AIC From a Gaussian Fit

A hypothetical regression analysis has five observations. Its fitted residuals are $1,-1,2,-2,0$. Suppose the fitted likelihood has $k=4$ free parameters. Calculate the maximized Gaussian log-likelihood contribution and AIC, assuming independent normal errors with an unknown common variance.

Method. First compute the residual sum of squares, then maximize the Gaussian likelihood over the positive variance, and finally apply the $2k$ correction.

The residual sum of squares is $$ \mathrm{RSS}=1^2+(-1)^2+2^2+(-2)^2+0^2=1+1+4+4+0=10. $$ Since $10>0$, the positive-variance maximum-likelihood estimate exists and is $$ \hat{\sigma}^2=\frac{\mathrm{RSS}}{n}=\frac{10}{5}=2. $$

Using natural logarithms, the maximized minus-two log likelihood is $$ -2\ell(\hat{\beta},\hat{\sigma}^2) =5\log(2\pi)+5+5\log(2) \approx 5(1.837877+1+0.693147) =17.6551. $$ Therefore, $$ \mathrm{AIC}=17.6551+2(4)=17.6551+8=25.6551. $$ Rounded to four decimal places, the AIC is $25.6551$. The calculation checks because the variance estimate is positive and the AIC exceeds the fit term by exactly $2k=8$. A common mistake is to add $k=4$ rather than $2k=8$; that would use the log-likelihood optimism on the wrong scale.

Known and Unknown Variance Are Different Likelihoods

The Gaussian fit term above assumes the variance is estimated. If $\sigma^2$ is known, it is not maximized over and must not be replaced by $\mathrm{RSS}/n$. For a fitted mean with residual sum of squares $\mathrm{RSS}$, the known-variance likelihood instead gives

-2\ell =n\log(2\pi\sigma^2)+\frac{\mathrm{RSS}}{\sigma^2}.

Only estimated parameters contribute to $k$. Thus, a known variance contributes no estimated variance parameter, while an unknown variance contributes one. Comparing criteria also requires using likelihoods defined for the same response data and a consistent likelihood convention; terms that are constant across candidate models do not change their ranking, but omitting a non-common term can change it.

Worked Example: Comparing Models With a Known Error Variance

A hypothetical analysis uses $n=3$ independent Gaussian observations with known error variance $\sigma^2=4$. Two candidate mean models have residual sums of squares $12$ and $8$, respectively. The first estimates $k=1$ mean parameter and the second estimates $k=2$. Calculate both AIC values and select the lower one.

Method. Because the variance is known, use $n\log(2\pi\sigma^2)+\mathrm{RSS}/\sigma^2$ for the fit term. Do not substitute the unknown-variance formula.

The common likelihood constant is $$ n\log(2\pi\sigma^2)=3\log(2\pi\cdot4)=3\log(8\pi) \approx 3(3.224171)=9.6725. $$ For the first model, the fit term and AIC are $$ -2\ell_1=9.6725+\frac{12}{4}=9.6725+3=12.6725, \qquad \mathrm{AIC}_1=12.6725+2(1)=14.6725. $$ For the second model, they are $$ -2\ell_2=9.6725+\frac{8}{4}=9.6725+2=11.6725, \qquad \mathrm{AIC}_2=11.6725+2(2)=15.6725. $$ Rounded to four decimal places, the AIC values are $14.6725$ and $15.6725$. The first model has the lower AIC, despite its larger residual sum of squares, because the second model’s improvement in fit does not offset its additional penalty. The check is that the fit term favors the second model by $1$, while its penalty is larger by $2$, so its AIC is larger by $1$. The trap is to use the unknown-variance expression even though the variance is specified as known.

When an Exact Fit Breaks the Calculation

The positive-residual-sum-of-squares condition is essential when the Gaussian variance is unknown. At an exact fit, reducing the variance concentrates the normal density around the fitted values and makes the likelihood arbitrarily large. This is not a case where one should substitute zero into a logarithm or report a finite AIC.

Worked Example: Detecting an Unbounded Gaussian Likelihood

A hypothetical regression has three observations, and its fitted mean equals every observed response exactly. Errors are assumed independent and normal with an unknown variance constrained to be strictly positive. Determine whether a finite maximum-likelihood estimate of the variance exists and whether the ordinary maximized-likelihood AIC is defined.

Method. Use the Gaussian likelihood and inspect its behavior as the positive variance approaches zero.

Exact fit means every residual is zero, so $\mathrm{RSS}=0$. The likelihood as a function of variance is $$ L(\sigma^2)=(2\pi\sigma^2)^{-3/2} \exp\!\left(-\frac{0}{2\sigma^2}\right) =(2\pi\sigma^2)^{-3/2}. $$ For any positive $\sigma^2$, this likelihood is finite, but $$ \lim_{\sigma^2\downarrow0}(2\pi\sigma^2)^{-3/2}=+\infty. $$ Therefore, no finite positive variance maximizes the likelihood. Since there is no finite maximum-likelihood likelihood value, the ordinary AIC based on that maximum is not defined for this case. The check is the unbounded limit, not a finite estimate of $0/3$. The common mistake is to mechanically use $\hat{\sigma}^2=\mathrm{RSS}/n=0$ in the AIC formula; that formula assumes $\mathrm{RSS}>0$.

Interpreting AIC Carefully

AIC balances maximized fit against an approximate predictive-optimism penalty. The penalty does not mean that every additional parameter is always harmful; a more complex model can still have lower AIC if its likelihood improvement is sufficiently large. Nor does the criterion establish that the selected model is correct. Its interpretation is comparative: among candidate models fitted to the same observations under comparable likelihood assumptions, lower AIC is preferred as an estimate of predictive performance.

Use a maximized likelihood. A likelihood evaluated at arbitrary parameter values is not the AIC fit term.
Use the appropriate variance assumption. The unknown-variance Gaussian expression does not apply when variance is fixed and known.
Use the full parameter count. The correction is $2k$, with $k$ counting free parameters estimated by the likelihood.
Check for regularity and existence. Boundary estimates, non-identifiability, or an unbounded likelihood can invalidate the usual derivation or calculation.
Compare like with like. Candidate likelihoods should describe the same observed response data on a consistent scale.

Key takeaway: In a regular likelihood model, fitting $k$ parameters produces about $k$ units of training log-likelihood optimism. On the minus-two-log-likelihood scale, AIC corrects this with $2k$. For Gaussian regression with an unknown positive variance and $\mathrm{RSS}>0$, maximizing the likelihood yields the fit term $n[\log(2\pi)+1+\log(\mathrm{RSS}/n)]$.

Check Your Understanding

Work each problem using the stated likelihood assumptions. Answers are included for checking.

A regular model has $k=6$ free parameters. Its maximized log likelihood is $-42$. Calculate the AIC and state the penalty added to the minus-two-log-likelihood fit term. (Answer: fit term $84$, penalty $12$, AIC $96$.)
A Gaussian regression has $n=6$, $\mathrm{RSS}=12$, and an unknown positive common variance. Its likelihood has $k=4$ free parameters. Calculate $\hat{\sigma}^2$ and the AIC, rounding the AIC to four decimal places. (Answer: $\hat{\sigma}^2=2$; AIC $=6[\log(2\pi)+1+\log(2)]+8\approx29.1861$.)
Two Gaussian mean models use $n=4$ observations and known variance $\sigma^2=2$. Model A has $\mathrm{RSS}=6$ and $k=1$; Model B has $\mathrm{RSS}=4$ and $k=2$. Calculate the AIC difference $\mathrm{AIC}_B-\mathrm{AIC}_A$ and select a model. (Answer: Model B’s fit improves by $1$, its penalty increases by $2$, so the difference is $+1$; select Model A.)
A Gaussian regression with unknown positive variance has $n=5$ and fits all observations exactly. Explain whether the usual variance MLE and maximized-likelihood AIC exist. (Answer: No finite positive variance MLE exists; the likelihood tends to infinity as $\sigma^2$ decreases to zero, so the ordinary AIC is not defined.)
In the regular-model derivation, the expected training log-likelihood gain from estimating $k$ parameters is approximately $k/2$, while the expected new-sample log-likelihood change from estimation is approximately $-k/2$. Find the training-over-prediction optimism in log-likelihood units and its corresponding AIC-scale penalty. (Answer: optimism $k$ log-likelihood units; AIC penalty $2k$.)

What You'll Learn