Why the Sample Must Be Held Fixed
Information criteria balance model fit against model complexity. For Gaussian linear regression, the fit component is based on the residual sum of squares, or RSS, and the penalty depends on the number of estimated parameters. But the comparison has a basic requirement: candidate models must explain the same response observations. Otherwise, their fit terms describe different data, so a lower criterion may reflect which cases were retained rather than a better model.
This issue commonly arises when candidate predictors have different patterns of missing values. A model using a predictor observed for everyone may be fitted to a larger sample than a model using another predictor with missing entries. Software that automatically deletes incomplete rows can then produce criteria based on different response records and different sample sizes. Those values should not be ranked as though they were calculated from one common likelihood.
Gaussian Regression Criteria and the Sample
Suppose the response values \(Y_1,\ldots,Y_n\) follow a Gaussian linear regression model. For candidate model \(j\), write \(q_j\) for its number of regression coefficients, including an intercept if present, and let \(RSS_j\) be its residual sum of squares. Assume the errors are independent and identically distributed as \(N(0,\sigma^2)\), the design matrix has full column rank, and \(RSS_j>0\). The Gaussian likelihood is maximized over the regression coefficients and the common error variance.
The total parameter count for AIC or BIC is \(k_j=q_j+1\): it includes the \(q_j\) regression coefficients and the estimated error variance. Under these assumptions, the maximum-likelihood estimate of the variance is \(RSS_j/n\). Thus, writing \(C=\log(2\pi)+1\), the criteria are:
These expressions are evaluated for a model fitted to \(n\) response values. When two models use the same response observations, \(n\) and the common terms cancel in a difference. If model 2 has \(k_2-k_1\) more parameters than model 1, then:
A negative difference favors model 2; a positive difference favors model 1. The formulas for a difference rely on common \(n\) and common response observations. If the samples differ, the cancellation that gives these simple comparisons is not available, and the likelihoods do not describe the same observed response vector.
Why the Common-Sample Condition Matters
The reason is visible from the Gaussian log-likelihood. Given a specific set of response values \(y_1,\ldots,y_n\), the log-likelihood is the sum of their log densities under the model. At the maximum-likelihood estimates, it is:
AIC and BIC combine this maximized log-likelihood with a complexity penalty: \(AIC=-2\ell_{\max}+2k\) and \(BIC=-2\ell_{\max}+k\log n\). The log-likelihood for one set of observed responses is not a measure of fit to another set. In particular, a criterion computed from fewer records cannot automatically be treated as directly comparable with one computed from more records. The criteria may have numerical values, but a numerical ranking does not by itself make the comparison meaningful.
The derivation of the variance estimate also shows why the sample size belongs in the formula. For fixed regression coefficients, the Gaussian log-likelihood contains the terms \(-n\log(\sigma^2)/2-RSS/(2\sigma^2)\). Differentiating with respect to \(\sigma^2\), setting the derivative to zero, and solving gives:
Consequently, changing the sample changes both the observations contributing to the RSS and the \(n\) used in the likelihood. The resulting criterion difference is not just a correction to the model complexity penalty.
Worked Example: Comparing Models on Twelve Common Responses
In a hypothetical portfolio study, two Gaussian regression models are fitted to the same 12 response records. Both include an intercept. Model 1 has no slope and has \(RSS_1=48\); model 2 adds one slope and has \(RSS_2=42\). Find the AIC and BIC differences and identify the preferred model under each criterion.
Method. Count the error variance as a parameter. Model 1 has one regression coefficient and a variance, so \(k_1=2\). Model 2 has two regression coefficients and a variance, so \(k_2=3\). The common sample size is \(n=12\), allowing use of the difference formulas.
For AIC:
Since \(\log(7/8)\approx-0.1335314\), the difference is \(-1.6023767+2=0.3976233\), or \(0.3976\) to four decimal places. It is positive, so model 1 has the lower AIC.
For BIC:
This is also positive, so model 1 has the lower BIC. The added slope reduces RSS, but the reduction is not large enough to offset the penalty under either criterion. The common mistake this method avoids is comparing RSS values alone or omitting the variance parameter from \(k\). As a check, both criteria favor the model with the smaller criterion difference, and both differences have the same positive sign.
Missing Predictors Can Change the Apparent Ranking
When predictors are missing, a standard complete-case fit uses only records with observed values for every variable in that model. If candidate models use different predictors, their complete-case sets may differ. Comparing the resulting AIC or BIC values can create an apparent ranking that changes when both models are refitted to the same cases.
The following hypothetical example makes this distinction explicit. Model A uses an intercept and predictor \(x_1\), which is observed for all ten records. Model B uses an intercept, \(x_1\), and predictor \(x_2\), which is observed for only six records. The response is available for all ten records. Thus, the model fits initially use different response observations.
Worked Example: A Missing Predictor Reverses the Apparent AIC Ranking
On its ten available response records, model A has \(RSS_A=40\). On the six complete records available for model B, model B has \(RSS_B=20\). Each model includes an intercept. Calculate the criteria as a diagnostic, explain why their direct ranking is not valid, and then refit both models on the same six records, where model A has \(RSS_A=25\).
Method. Model A has two regression coefficients plus the error variance, so \(k_A=3\). Model B has three regression coefficients plus the error variance, so \(k_B=4\). Use the Gaussian AIC formula, with \(C=\log(2\pi)+1\approx2.8378771\).
The AIC calculated for model A on its ten records is:
The AIC calculated for model B on its six records is:
A mechanical ranking would say model B has the lower AIC. That conclusion is not justified: model A was assessed on ten responses, while model B was assessed on six different responses. The criteria refer to different observed response sets.
Now restrict model A to the same six records used by model B, and use its refitted \(RSS_A=25\). Its common-sample AIC is:
Model B remains at \(32.2511\) on those same six responses. The valid common-sample difference is:
The positive difference means model A has the lower AIC on the common sample, despite model B's smaller RSS. Thus, the apparent preference for model B in the first calculation is not a valid comparison, and the common-sample refit gives the opposite ranking. The common mistake is treating a smaller criterion from a smaller, different sample as decisive. As a check, the common-sample comparison uses \(n=6\) in both fit terms and counts one additional parameter for model B.
A Second Common-Sample Calculation
A common sample is necessary, but it does not guarantee that the more complex model wins. Its improvement in fit must still outweigh its additional penalty. The parameter count must also be consistent: under the stated Gaussian model it includes the regression coefficients and the error variance.
Worked Example: Counting Parameters in a Six-Response Comparison
Two Gaussian regression models use the same six hypothetical response records. Model 1 has an intercept and one slope, with \(RSS_1=64\). Model 2 adds a second slope, with \(RSS_2=55\). Find each AIC and determine which is lower.
Method. The regression coefficient counts are two and three, respectively. Including the estimated error variance gives \(k_1=3\) and \(k_2=4\). With \(n=6\) and \(C=2.8378771\), calculate each criterion.
The difference is \(AIC_2-AIC_1=38.3207-37.2300=1.0907\), so model 1 has the lower AIC. Equivalently, the difference formula gives \(6\log(55/64)+2=1.0907\). Model 2 has a lower RSS, but the gain is insufficient to offset the two-unit AIC penalty for its extra parameter. This calculation avoids the mistake of using a different denominator in the two models: both use \(RSS/n\) with \(n=6\). The direct subtraction and the difference formula agree to four decimal places.
Practical Checks Before Ranking Models
Before interpreting an AIC or BIC ranking, check what observations and response values went into each fit. A concise audit can prevent a technically correct calculation from supporting an invalid conclusion.
Confirm that each candidate model uses the same observational units and the same response values, not merely the same number of rows.
For the Gaussian formulas here, the response must be modeled on the same scale under the same likelihood setup. A numerical criterion from a different response representation should not be silently treated as directly comparable.
For predictor-related missingness, one direct option is to define a common complete-case set using the response and all candidate predictors, then refit every candidate model to those records.
For Gaussian regression, count the regression coefficients and the error variance. Then compare criteria calculated with the same \(n\) and response observations.
Using one common complete-case set makes the likelihood comparison consistent, but it may discard useful observations. It also does not establish that missingness is harmless for the broader analysis. The specific point here is narrower: if models have been fitted to different response records, their information criteria do not provide the intended like-for-like ranking.
Check Your Understanding
For each question, verify both the sample condition and the arithmetic before interpreting the ranking.
- Two Gaussian models use the same 15 response observations. Model 1 has \(k_1=3\) and \(RSS_1=60\); model 2 has \(k_2=4\) and \(RSS_2=48\). Calculate \(AIC_2-AIC_1\) to four decimal places and state which model has the lower AIC. (Answer: \(-1.3472\); model 2.)
- Two models use the same eight responses. Model 1 has \(RSS_1=32\), and model 2 adds one parameter and has \(RSS_2=24\). Find \(AIC_2-AIC_1\) to four decimal places and identify the preferred model. (Answer: \(-0.3015\); model 2.)
- Two models use the same 20 responses. Model 1 has \(k_1=4\), \(RSS_1=100\); model 2 has \(k_2=6\), \(RSS_2=90\). Calculate \(AIC_2-AIC_1\) to four decimal places. (Answer: \(1.8928\); model 1 has the lower AIC.)
- Model A is fitted to 12 response records with \(RSS_A=48\), while model B is fitted to eight different complete-case records with \(RSS_B=28\). Can their AIC values be ranked directly as evidence about which model fits better? (Answer: No; they use different response observations, so refit on a common sample before comparing.)
- On the same eight response records, model A has \(k_A=3\), \(RSS_A=36\), and model B has \(k_B=4\), \(RSS_B=28\). Calculate \(AIC_B-AIC_A\) to four decimal places. (Answer: \(-0.0105\); model B has the slightly lower AIC.)