Counting Parameters Correctly in Information Criteria

Why Parameter Counting Matters

Information criteria balance a model’s fit to the observed data against a penalty for the number of parameters estimated. Counting too few parameters weakens that penalty and can make a model look artificially attractive. The most common source of confusion is that a regression model estimates both its mean coefficients and an error-variance parameter.

This tutorial uses the ordinary Gaussian regression setting and makes the parameter count explicit. It also distinguishes a full formula, which retains the likelihood constants and counts every estimated parameter, from a reduced formula, which removes constants shared by all models being compared. The two forms give the same ranking only when the omitted terms really are common to those models.

Count Coefficients and the Error Variance

Suppose a model is fitted to \(n\) observations using a response \(Y\) and a design matrix \(X\) of full column rank. Write the model as

Y_i=\beta_0+\beta_1x_{i1}+\cdots+\beta_{k-1}x_{i,k-1}+\varepsilon_i,\qquad \varepsilon_i\overset{\mathrm{iid}}{\sim}N(0,\sigma^2).

Here \(k\) is the number of regression coefficients estimated, including the intercept if the model has one. The error variance \(\sigma^2\) is another parameter estimated from the data. Therefore, the total number of estimated likelihood parameters is \(K=k+1\). This tutorial assumes \(\sigma^2\) is unknown and estimated.

Definition: In Gaussian regression with an unknown error variance, \(k\) counts the estimated regression coefficients and \(K=k+1\) counts all estimated likelihood parameters: the \(k\) coefficients plus the single error-variance parameter. Use \(K\) in the full AIC, AICc, and BIC penalty formulas.

Count coefficients, not just the names of predictors. A categorical predictor with \(g\) levels is typically represented by \(g-1\) indicator or contrast coefficients when an intercept is included. An interaction may add several coefficients, depending on the variables involved. If the model matrix has redundant columns, the number of independently estimable coefficients is its rank; counting redundant terms as separate estimated coefficients would be incorrect.

The setup also specifies the assumptions behind the formulas used here: the mean is linear in the included coefficients, errors are independent and normally distributed, and all observations share the same error variance. The candidate models must be fitted to the same response observations for the reduced-form comparisons below to be equivalent.

Worked Example: Counting a Categorical Predictor

A hypothetical model includes an intercept, two numerical predictors, and a categorical predictor with three levels. The categorical predictor is represented using two indicator coefficients. The response errors are assumed independent and normally distributed with one common, unknown variance. Find \(k\) and \(K\).

Regression coefficients. There is one intercept, two numerical-predictor slopes, and two indicator coefficients. Thus, \(k=1+2+2=5\).

Total estimated parameters. The unknown error variance adds one parameter, so \(K=k+1=5+1=6\). The category variable contributes two coefficients, not three: with an intercept, one level is the reference level.

If a criterion’s full penalty calls for \(K\), use 6. Using 5 would count the regression coefficients but omit the estimated variance.

Full Likelihood Formulas

For the Gaussian model, let \(\mathrm{RSS}\) be the residual sum of squares. The maximum-likelihood estimate of the error variance is \(\widehat{\sigma}^2=\mathrm{RSS}/n\), not the unbiased estimate \(\mathrm{RSS}/(n-k)\). Substituting the maximum-likelihood estimate into the normal log-likelihood gives the full fit term used in the criteria.

Result: For Gaussian regression with an unknown variance estimated by maximum likelihood, the full criteria are

\begin{aligned} \mathrm{AIC} &=n\left[\ln(2\pi)+1+\ln\left(\frac{\mathrm{RSS}}{n}\right)\right]+2K,\\[4pt] \mathrm{AICc} &=\mathrm{AIC}+\frac{2K(K+1)}{n-K-1},\\[4pt] \mathrm{BIC} &=n\left[\ln(2\pi)+1+\ln\left(\frac{\mathrm{RSS}}{n}\right)\right]+K\ln(n). \end{aligned}

A smaller criterion value is preferred among the models being compared. For the AICc formula shown here, the denominator must be positive: \(n-K-1>0\). If that condition fails, this AICc formula is not defined. Do not silently substitute a different parameter count or treat an undefined value as a model score.

The \(K\) in the AICc correction is the total parameter count, including the variance. The correction is not just another fixed penalty per regression coefficient: its size depends on \(K\) and \(n\). BIC likewise uses the total \(K\) in its full penalty.

Why the Full AIC Fit Term Has This Form

A short likelihood calculation shows where the full fit term comes from. Under the stated independent normal-error assumptions, the log-likelihood for a model with mean values \(\mu_i\) is

\ell(\boldsymbol{\beta},\sigma^2) =-\frac{n}{2}\ln(2\pi\sigma^2) -\frac{1}{2\sigma^2}\sum_{i=1}^n(Y_i-\mu_i)^2.

At the maximum-likelihood estimates, the sum of squared residuals is \(\mathrm{RSS}\), and maximizing over \(\sigma^2\) gives \(\widehat{\sigma}^2=\mathrm{RSS}/n\). Substitute this estimate into the log-likelihood:

\begin{aligned} \ell_{\max} &=-\frac{n}{2}\ln\left(2\pi\frac{\mathrm{RSS}}{n}\right) -\frac{\mathrm{RSS}}{2(\mathrm{RSS}/n)}\\ &=-\frac{n}{2}\left[\ln(2\pi)+\ln\left(\frac{\mathrm{RSS}}{n}\right)+1\right]. \end{aligned}

Thus \(-2\ell_{\max}=n[\ln(2\pi)+1+\ln(\mathrm{RSS}/n)]\). AIC adds \(2K\) to this quantity; BIC adds \(K\ln(n)\). AICc adds its stated finite-sample correction to AIC.

When Reduced Formulas Are Equivalent

For models fitted to the same \(n\) observations and the same response, the terms \(n[\ln(2\pi)+1]\) are shared. They may be removed when ranking models. The variance parameter is also counted in the full parameter penalty, even when a reduced expression removes the corresponding common penalty contribution.

\begin{aligned} \mathrm{AIC}_{\mathrm{red}} &=n\ln\left(\frac{\mathrm{RSS}}{n}\right)+2k,\\ \mathrm{AICc}_{\mathrm{red}} &=n\ln\left(\frac{\mathrm{RSS}}{n}\right)+2k +\frac{2K(K+1)}{n-K-1},\qquad K=k+1,\\ \mathrm{BIC}_{\mathrm{red}} &=n\ln\left(\frac{\mathrm{RSS}}{n}\right)+k\ln(n). \end{aligned}

Each reduced expression is the corresponding full criterion minus a constant common to the candidate models. For AIC, the removed constant is \(n[\ln(2\pi)+1]+2\); for BIC, it is \(n[\ln(2\pi)+1]+\ln(n)\). The AICc correction still uses \(K=k+1\), even though its reduced formula writes \(2k\) for the AIC penalty. Dropping the variance parameter from the correction would change the comparison rather than merely remove a shared constant.

Worked Example: Comparing Two Models With AIC

Two hypothetical Gaussian regressions use the same \(n=40\) observations and response. Model A has an intercept and one slope, with \(\mathrm{RSS}=400\). Model B has an intercept and three slopes, with \(\mathrm{RSS}=300\). Assume independent normal errors with a common unknown variance in each model. Compare their AIC values.

For Model A, \(k_A=2\) and \(K_A=3\). For Model B, \(k_B=4\) and \(K_B=5\). Using the reduced AIC, the difference is

\begin{aligned} \mathrm{AIC}_{B,\mathrm{red}}-\mathrm{AIC}_{A,\mathrm{red}} &=40\ln\left(\frac{300/40}{400/40}\right)+2(4-2)\\ &=40\ln(0.75)+4\\ &\approx -11.507+4\\ &=-7.507. \end{aligned}

Model B has the smaller AIC by about 7.507, so it is preferred by AIC among these two models. This comparison includes the cost of the additional regression coefficients. Both models also estimate a variance parameter; that common contribution is already handled by using the reduced formula consistently.

Worked Example: Calculating AICc With the Full Count

A hypothetical regression is fitted to \(n=30\) observations. It has an intercept, two numerical-predictor slopes, and a three-level categorical predictor represented by two coefficients, giving \(k=5\). Suppose \(\mathrm{RSS}=240\), and assume the Gaussian model conditions used above. Calculate the full AIC and AICc.

The total parameter count is \(K=k+1=6\). First calculate the full AIC:

\begin{aligned} \mathrm{AIC} &=30\left[\ln(2\pi)+1+\ln\left(\frac{240}{30}\right)\right]+2(6)\\ &=30[1.837877+1+\ln(8)]+12\\ &\approx 30(4.91732)+12\\ &\approx 159.520. \end{aligned}

The AICc denominator is \(n-K-1=30-6-1=23\), which is positive. Its correction is

\frac{2K(K+1)}{n-K-1} =\frac{2(6)(7)}{23} =\frac{84}{23} \approx 3.652.

Therefore, \(\mathrm{AICc}\approx159.520+3.652=163.172\). The calculation uses six, not five, in the correction because the error variance is an estimated likelihood parameter.

Count Model Structure, Not Just Predictor Names

Interactions and categorical predictors make a useful check on whether a count is being done correctly. For example, a model with an intercept, one numerical predictor, one binary-group indicator, and their interaction has four regression coefficients: an intercept, two main-effect coefficients, and one interaction coefficient. It therefore has \(K=5\) when the error variance is unknown.

Worked Example: Checking an AICc Denominator

A hypothetical model uses \(n=12\) observations and contains an intercept, a numerical predictor, a binary-group indicator, and their interaction. Assume the Gaussian regression conditions and an unknown error variance. Determine its AICc correction and check that the formula is defined.

The four regression coefficients give \(k=4\), so \(K=k+1=5\). The denominator is \(n-K-1=12-5-1=6\), which is positive. Thus,

\frac{2K(K+1)}{n-K-1} =\frac{2(5)(6)}{6} =10.

The AICc correction is 10. If one mistakenly used \(k=4\) in place of \(K\), the result would be \(2(4)(5)/(12-4-1)=40/7\approx5.714\), which is not the correction for the stated full parameter count.

Common Mistakes and a Practical Check

Omitting the error variance. Wrong reasoning: “There are five coefficients, so \(K=5\).” Correction: if the error variance is unknown and estimated, it adds one likelihood parameter, so \(K=6\).
Counting a categorical variable as one coefficient. Wrong reasoning: “The model has one categorical predictor, so it adds one parameter.” Correction: count the coefficients used to represent its levels. With an intercept, a \(g\)-level factor commonly adds \(g-1\) coefficients.
Using \(k\) in the AICc correction. The reduced AIC penalty may be written \(2k\), but the AICc correction uses \(K=k+1\) under the unknown-variance convention in this tutorial.
Mixing full and reduced formulas. A reduced fit term cannot be paired with an unadjusted full-form penalty and then treated as the same criterion. Use one complete form consistently.
Comparing scores after removing nonshared terms. A constant is safe to omit only if it is the same for every model in the comparison. Here the reduced forms are justified for candidates fitted to the same response observations.

A reliable calculation therefore proceeds in order: count the independently estimated regression coefficients, add the estimated error variance to obtain \(K\), select the full or reduced form, and check the AICc denominator before calculating its correction. This makes the model-complexity penalty transparent rather than an unexplained number in a software output.

Key takeaway: For Gaussian regression with an unknown error variance, \(k\) counts regression coefficients and \(K=k+1\) counts all estimated likelihood parameters. Full AIC, AICc, and BIC use \(K\); equivalent reduced formulas may omit shared terms, but AICc’s correction must still use the full \(K\).

Check Your Understanding

Answer each question by stating your parameter count and reasoning.

A model has an intercept, three numerical slopes, and a four-level categorical predictor represented with an intercept. What are \(k\) and \(K\) if the error variance is unknown?
Why may the full AIC and reduced AIC give different numerical values but the same ranking for models fitted to the same observations?
A model has \(k=3\) regression coefficients and \(n=9\). Calculate \(K\) and determine whether the AICc formula in this tutorial has a positive denominator.
In the reduced AICc formula, which parameter count belongs in the small-sample correction: \(k\) or \(K\)? Explain.
A predictor is described as one categorical variable with five levels. Why is its coefficient count generally not five when the model also includes an intercept?