Deriving The Small-Sample Correction AICc

Why Correct AIC in a Small Sample?

AIC adjusts a model’s maximized likelihood to account for the optimism that comes from fitting parameters to the observed data. Its familiar penalty is $2K$, where $K$ is the number of estimated likelihood parameters. That penalty is motivated by an asymptotic argument: it is most reliable when the sample is large relative to the model’s complexity. With a small sample, fitting can create more optimism than $2K$ captures, so a further correction is useful.

AICc adds a finite-sample term to AIC. The standard form is often used for regular likelihood models as a second-order correction, but it is not an exact universal correction for every distribution or sampling design. A useful exact derivation is available for Gaussian linear regression with an estimated variance. It shows both where the correction comes from and why the sample-size condition matters.

Definition: For a fitted model with maximized log likelihood $\hat{\ell}$, sample size $n$, and $K$ estimated likelihood parameters, define $$ \mathrm{AIC}=-2\hat{\ell}+2K $$ and the standard small-sample correction $$ \mathrm{AICc}=\mathrm{AIC}+\frac{2K(K+1)}{n-K-1}. $$ Here $K$ counts all estimated likelihood parameters under a consistent convention, including an estimated variance parameter. The correction is defined only when $n-K-1>0$, or equivalently $n>K+1$.

In the Gaussian regression derivation below, $n$ is the number of independent observations and $K$ includes the regression coefficients and the error variance. For other regular likelihood models, the same formula is a commonly used finite-sample approximation; its accuracy depends on the model and sampling assumptions. A model-specific correction may be preferable when one is available.

Deriving the Correction in Gaussian Regression

Assume a correctly specified normal linear model with fixed design matrix $X$, full column rank $p$, and independent errors with common variance:

Y=X\beta+\varepsilon,\qquad \varepsilon\sim N(0,\sigma^2 I_n).

The $p$ regression coefficients and the variance $\sigma^2$ are estimated, so $K=p+1$. The maximum-likelihood estimate of the variance is $\hat{\sigma}^2=\mathrm{RSS}/n$, where $\mathrm{RSS}$ is the residual sum of squares. To assess predictive optimism, compare the fitted deviance on the training observations with the expected deviance on an independent replicate of $n$ observations at the same design points.

Find the training residual degrees of freedom.
Under the stated model, $\mathrm{RSS}/\sigma^2\sim\chi^2_\nu$, where $\nu=n-p$. The fitted variance is $\mathrm{RSS}/n$.

Calculate the expected predictive quadratic term.
For a replicate response vector, the expected residual sum of squares, averaged over training data and the replicate, is $n\sigma^2+p\sigma^2=(n+p)\sigma^2$. The first contribution is replicate error; the second is the prediction error from estimating $p$ coefficients.

Compare predictive and fitted deviance.
The log-variance terms have the same expectation in the predictive and fitted deviance because both use the training estimate $\mathrm{RSS}/n$. The difference comes from the quadratic terms; evaluating it gives the finite-sample optimism.

Here is the calculation in full. The fitted training negative log likelihood is $\frac n2\log(2\pi\mathrm{RSS}/n)+\frac n2$, since the fitted residual sum of squares divided by $\hat{\sigma}^2$ equals $n$. For the independent replicate, conditional on the fitted coefficients and variance estimate, its expected negative log likelihood is $\frac n2\log(2\pi\mathrm{RSS}/n)+\frac{n\sigma^2+\|X(\hat{\beta}-\beta)\|^2}{2(\mathrm{RSS}/n)}$. Under the stated normal model, $\hat{\beta}$ is independent of $\mathrm{RSS}$, and $E[\|X(\hat{\beta}-\beta)\|^2]=p\sigma^2$. Also, for $U\sim\chi^2_\nu$, $E[1/U]=1/(\nu-2)$ when $\nu>2$. Thus, when $n-p>2$, the expected difference in negative log likelihood is

\frac{1}{2}\left(\frac{n(n+p)}{n-p-2}-n\right) =\frac{n(p+1)}{n-p-2}.

Deviance is negative two times log likelihood, so the corresponding expected difference in deviance is twice this value. Since $K=p+1$, and $n-p-2=n-K-1$, the expected predictive deviance minus the expected fitted deviance is

\frac{2nK}{n-K-1} =2K+\frac{2K(K+1)}{n-K-1}.

The right-hand expression separates the usual AIC penalty $2K$ from the additional small-sample correction. Adding the latter to AIC gives the stated AICc formula. This derivation is exact for the specified Gaussian regression scoring setup when $n>K+1$; it does not establish an exact correction for every regular likelihood model.

Applying AICc and Checking Its Conditions

The formula is straightforward once the parameter count is settled. The harder exam task is often deciding what belongs in $K$. Count parameters estimated by maximizing the likelihood, not just the regression slopes or the parameters that are most visible in a model description. For a normal regression with an intercept, two slopes, and an estimated error variance, $K=4$. If a parameter is fixed rather than estimated, it is not counted as an estimated likelihood parameter.

Worked Example: Correcting AIC for a Small Regression Sample

A hypothetical normal linear regression is fitted to $n=32$ independent observations. It estimates an intercept, two slopes, and a common error variance. Its AIC is 145. Calculate AICc to two decimal places and check the sample-size condition.

Method and parameter count. There are three regression coefficients and one estimated variance parameter, so $K=3+1=4$. The denominator condition is $32-4-1=27>0$, so AICc is defined.

Correction. $$ \frac{2K(K+1)}{n-K-1} =\frac{2(4)(5)}{32-4-1} =\frac{40}{27} =1.481481\ldots $$

Corrected criterion. Therefore, $$ \mathrm{AICc}=145+\frac{40}{27}=146.481481\ldots=146.48. $$

The answer is 146.48 to two decimal places. The common mistake avoided is counting only the three regression coefficients: the estimated variance is also a likelihood parameter.

A positive denominator is a mathematical requirement, not a guarantee of practical reliability. As $n-K-1$ approaches zero, the correction becomes large and highly sensitive to the parameter count. In that situation, AICc may be defined but still provide a poor basis for confident model selection. In the Gaussian derivation, the condition $n>K+1$ also ensures the inverse-residual expectation used in the calculation exists.

Worked Example: Defined Does Not Mean Stable

A hypothetical likelihood model fitted to $n=14$ independent observations estimates $K=12$ parameters and has AIC $=80$. Determine whether AICc is defined, calculate it, and assess the size of the correction.

Method. Check the denominator first: $$ n-K-1=14-12-1=1. $$ It is positive, so the formula is defined.

Correction. $$ \frac{2K(K+1)}{n-K-1} =\frac{2(12)(13)}{1} =312. $$

Corrected criterion. $$ \mathrm{AICc}=80+312=392. $$

AICc is defined and equals 392. However, the sample size is only slightly larger than the parameter count, so the correction is very large and the result is likely to be unstable. The common mistake avoided is assuming that a positive denominator alone makes the criterion practically reliable.

Using the Correction to Compare Models

When candidate models use the same observations and likelihood framework, calculate each model’s AICc using its own $K$, then compare the resulting values; a smaller value is preferred by this criterion. A model with a lower AIC can lose that advantage after correction if it uses more parameters and the sample is small. The correction is not a probability that a model is true, nor does AICc by itself establish that any candidate model is adequate.

Worked Example: AIC Ranking Reverses After Correction

Two hypothetical regular likelihood models are fitted to the same $n=30$ independent observations. Model A has $K=3$ and AIC $=120$; Model B has $K=5$ and AIC $=118$. Calculate both AICc values to two decimal places and identify the lower value.

Model A. Its denominator is $30-3-1=26$, which is positive. Its correction is $$ \frac{2(3)(4)}{26}=\frac{24}{26}=0.923076\ldots. $$ Thus $\mathrm{AICc}_A=120+0.923076\ldots=120.92$.

Model B. Its denominator is $30-5-1=24$, also positive. Its correction is $$ \frac{2(5)(6)}{24}=\frac{60}{24}=2.5. $$ Thus $\mathrm{AICc}_B=118+2.5=120.50$.

Model B has the lower AICc, 120.50 versus 120.92, although its advantage is small. The common mistake avoided is comparing the AIC values alone after the task asks for a small-sample comparison.

Common Exam Traps and Practical Limits

Using the wrong count. State what is estimated and count those parameters consistently. In a normal regression with unknown variance, omitting the variance from $K$ changes both the numerator and denominator.
Checking only whether $n$ is positive. AICc requires $n-K-1>0$. If the denominator is zero or negative, the standard formula is undefined; do not report a finite corrected value.
Calling a barely defined value reliable. A small positive denominator can produce an enormous correction. Report that limitation rather than treating the arithmetic result as decisive.
Applying the formula as an exact universal result. The derivation above uses normal linear regression, independent observations, a correctly specified model, fixed full-rank design, and an estimated common variance. In other regular likelihood settings, the standard formula is an approximation, and assumptions or model-specific corrections matter.
Adding the correction to the wrong quantity. AICc is AIC plus $2K(K+1)/(n-K-1)$; it is not the correction by itself and does not replace the $2K$ already included in AIC.

Key takeaway: AICc adds a parameter-dependent finite-sample correction to AIC. Count all estimated likelihood parameters consistently, verify $n>K+1$, and distinguish the exact Gaussian-regression derivation from the standard approximation used for other regular likelihood models.

Check Your Understanding

For each problem, use the stated $K$ as the total number of estimated likelihood parameters. Round corrected criteria to two decimal places where needed.

A model fitted to $n=50$ observations has $K=4$ and AIC $=198$. Calculate AICc. (Answer: 198.89)
A hypothetical model has $n=20$, $K=5$, and AIC $=96$. Calculate AICc. (Answer: 100.29)
A model has $n=11$ and $K=10$. Is the standard AICc formula defined? (Answer: No; $n-K-1=0$.)
Two models fitted to the same $n=80$ observations have the following values: Model A has $K=3$, AIC $=240$; Model B has $K=6$, AIC $=238$. Which has the lower AICc? (Answer: Model B; AICc values are 240.32 for A and 239.15 for B.)
A model fitted to $n=9$ observations has $K=2$ and AIC $=40$. Calculate AICc and state whether the denominator condition holds. (Answer: Defined; $n-K-1=6$, correction $=2.00$, and AICc $=42.00$.)

What You'll Learn