Counting Parameters In Regression AIC

Why Parameter Counting Matters in Regression AIC

AIC balances a model’s maximized likelihood against a penalty for the number of estimated parameters. In Gaussian linear regression, that count includes more than the slopes: it also includes an intercept when one is estimated and the error variance when it is unknown. A one-parameter counting difference changes the penalty by 2, so a small bookkeeping error can change a model comparison.

The key is to count free, identifiable parameters in the likelihood, rather than simply counting predictor names. The rank of the design matrix determines the number of distinct regression coefficients. The error variance is another parameter if it is estimated. An intercept is already among the regression coefficients; it is not an extra parameter to add after counting all coefficient terms.

This statement assumes a Gaussian model with a common, unknown variance. If the variance is specified as known, it is not estimated and is not counted. If a model has additional free distributional parameters, they must also be counted. A coefficient constrained by the model is not a free parameter; an unidentifiable coefficient does not add a distinct likelihood parameter. In the usual full-rank regression design, \(q\) is the number of columns of \(X\).

The Gaussian Likelihood and the Count

For the model above, the residual sum of squares at the least-squares coefficient estimate is \(\mathrm{RSS}=\sum_{i=1}^{n}(y_i-\widehat{y}_i)^2\). When \(\mathrm{RSS}>0\), maximizing the Gaussian likelihood over the coefficients and the unknown variance gives \(\widehat{\beta}\) as a least-squares estimate and the maximum-likelihood variance estimate \(\widehat{\sigma}^2_{\mathrm{ML}}=\mathrm{RSS}/n\). The denominator is \(n\), not the residual degrees of freedom \(n-q\).

For \(\mathrm{RSS}>0\), the maximized negative twice log-likelihood and the full AIC are therefore

The formula requires a finite maximum-likelihood estimate of the variance, which in this setting requires \(\mathrm{RSS}>0\). A saturated model whose design matrix has rank \(n\) fits every observation exactly, so \(\mathrm{RSS}=0\). More generally, any exact fit with zero RSS creates the same issue: the Gaussian likelihood increases without bound as \(\sigma^2\) approaches zero. There is then no finite maximum-likelihood estimate of the variance, and the displayed log-likelihood and ordinary AIC calculation are not finite.

Why the Variance Counts

The likelihood depends on both the mean parameters and the error variance. For a fixed coefficient vector, its log-likelihood is

The first term depends on \(\sigma^2\), and the second term depends on it as well. Thus, when the variance is unknown, choosing its value is part of fitting the model, just as choosing \(\beta\) is. That is why the variance contributes one estimated parameter to the full AIC count.

For completeness, fix \(\beta=\widehat{\beta}\) and write the residual sum of squares as \(R>0\). Differentiating the log-likelihood with respect to \(\sigma^2\) gives

Setting this derivative to zero gives \(R=n\sigma^2\), hence \(\widehat{\sigma}^2_{\mathrm{ML}}=R/n\). The second derivative at this value is negative, so it is a maximum. Substitution into the log-likelihood produces the stated expression, provided \(R>0\). If \(R=0\), this calculation has no positive finite solution; the likelihood is unbounded as the variance tends to zero.

What Counts as a Regression Coefficient?

A predictor name does not necessarily correspond to one coefficient. For example, a categorical predictor with four levels is often represented by three indicator columns when a reference level is used. Those columns contribute three coefficients, not one. A continuous predictor represented by one column contributes one coefficient. An interaction or other model term contributes as many independent columns as its parameterization requires.

The reliable count is the rank \(q\) of the design matrix, not the number of labels in a model formula. This also handles redundant columns: if one column is an exact linear combination of others, the design matrix rank does not increase, and it does not create another identifiable coefficient. In the common case of a full-column-rank design, count the intercept column if present, then count each additional independent coefficient column.

Reporting Conventions and Common Traps

Some regression summaries report a version of Gaussian AIC with a likelihood constant omitted. Because \(n\log(2\pi)\) is the same for models fitted to the same response observations, a commonly used reduced expression is

This is the full AIC minus \(n\log(2\pi)\). It gives the same ranking among models fit to the same observations, but its reported numerical value is not the full AIC value. When comparing reported values, identify whether the constant is included and whether the parameter count includes the variance. A convention that omits the variance from \(k\) also changes the value by 2 per model relative to the full count; it should not be mistaken for the full likelihood-based AIC defined here.

Another common error is substituting the unbiased estimate \(s^2=\mathrm{RSS}/(n-q)\) into the maximized-likelihood formula. That estimate is often useful for other regression purposes, but it is not the maximum-likelihood estimate used to obtain this Gaussian AIC expression. Also check whether the model includes an intercept, whether its design columns are independent, and whether the candidate models use the same observations. A comparison with different samples or different reporting conventions is not a straightforward comparison of the displayed AIC values.

A particularly serious trap is mechanically applying the formula when \(\mathrm{RSS}=0\). For example, if \(q=n\) and \(X\) has rank \(n\), the fitted values can equal every observed response, yielding \(\mathrm{RSS}=0\). Substituting this into \(\log(\mathrm{RSS}/n)\) produces \(\log(0)\), not a finite AIC. The issue is not a rounding problem: the Gaussian likelihood has no finite maximum over the unknown variance.