Prediction Criteria Are Not Test-Set Error
A regression criterion can reward a model that predicts well without measuring its error on new observations directly. Adjusted R-squared, Mallows’ Cp, AIC, and BIC use the fitted data and a complexity adjustment to assess competing models. A test-set error, by contrast, is calculated from outcomes deliberately held out from fitting. Understanding this distinction helps explain why the methods can rank models differently.
The criteria do not all target the same quantity. Some are connected to expected prediction error under a specified model; others are primarily measures of relative information loss or model evidence. A favorable criterion is therefore evidence for a model under that criterion’s assumptions—not a promise of future accuracy.
What the Criteria Measure
For the formulas below, suppose there are \(n\) observations and candidate Gaussian linear regression models fitted to the same response values on the same observations. Let \(RSS\) be a model’s residual sum of squares, \(TSS=\sum_{i=1}^n(y_i-\bar y)^2\), and \(q\) the number of estimated mean-regression coefficients, including the intercept. The Gaussian error variance is also estimated; because it is common to the candidate models, its contribution to AIC or BIC can be omitted when comparing those models.
Mallows’ \(C_p\): Given an estimate \(s^2\) of the error variance from a sufficiently rich reference model, \(C_p=RSS/s^2-n+2q\). It is a prediction-error-oriented criterion under linear-model assumptions; smaller values are generally preferred, while values near the model’s parameter count can indicate limited estimated bias.
AIC and BIC: For Gaussian linear regression, comparison forms that omit constants common to all candidate models are \(AIC=n\log(RSS/n)+2q\) and \(BIC=n\log(RSS/n)+q\log n\). Lower values are preferred within the comparison. The full likelihood counts the variance parameter too, but its penalty is common when every model estimates that same variance.
Adjusted R-squared penalizes the residual sum of squares through its residual degrees of freedom. AIC and BIC instead add explicit penalties to a likelihood-based measure of fit. AIC is motivated by estimating relative expected information loss, and in this Gaussian setting it is connected to expected predictive performance. BIC’s penalty arises from an approximation to model evidence; it is not designed as a direct estimate of test-set prediction error. The earlier material on AIC and BIC explains their penalty structures and model-comparison interpretation.
| Measure | What it is connected to | What it does not directly provide |
|---|---|---|
| Adjusted R-squared | Fit adjusted for regression degrees of freedom | Observed error on an independent test set |
| Mallows’ \(C_p\) | Estimated prediction error under linear-model conditions | A guarantee that the selected model predicts best in future data |
| AIC | Estimated relative expected information loss; related predictive assessment | The realized prediction error on a particular held-out sample |
| BIC | An approximation related to model evidence | A direct estimate of predictive error or a test-set score |
| Test-set error | Loss observed on the held-out observations | Exact future error; its estimate varies with the test sample |
Why Mallows’ \(C_p\) Has a Prediction Interpretation
A short derivation shows the source of the \(2q\) adjustment. Assume the fixed-design Gaussian linear model \(Y=\mu+\varepsilon\), where the errors are independent with mean zero and variance \(\sigma^2\). For a candidate regression model with projection matrix \(H\) and rank \(q\), fitted values are \(HY\). First suppose the candidate model contains the true mean vector, so \(H\mu=\mu\).
The residual vector is \((I-H)Y=(I-H)\varepsilon\). Since \(I-H\) is a projection of rank \(n-q\), \(E[RSS]=E[\varepsilon^{T}(I-H)\varepsilon]=(n-q)\sigma^2\).
The prediction error for the mean is \(HY-\mu=H\varepsilon\). Since \(H\) has rank \(q\), \(E[\|HY-\mu\|^2]=E[\varepsilon^{T}H\varepsilon]=q\sigma^2\).
Adding \(2q\sigma^2\) to training \(RSS\), then subtracting the common baseline \(n\sigma^2\), has expectation \((n-q)\sigma^2+2q\sigma^2-n\sigma^2=q\sigma^2\), the expected squared error in estimating the mean.
Dividing the corrected quantity by \(\sigma^2\) gives \(RSS/\sigma^2-n+2q\), the form of \(C_p\). In practice, \(\sigma^2\) is replaced by a reference estimate \(s^2\). If future outcomes, rather than their conditional means, are being predicted, their irreducible error variance adds a common contribution when the models face the same prediction setting. The derivation explains the correction under its assumptions; it does not make \(C_p\) an exact score for every data-generating process.
Worked Example: Adjusted R-squared and \(C_p\)
A hypothetical insurer models a response using \(n=30\) observations. The total sum of squares is \(TSS=200\). A smaller model has \(RSS=120\) and \(q=3\) coefficients; a larger model has \(RSS=100\) and \(q=5\). Use the larger model as the reference for the error variance. Its residual degrees of freedom are \(30-5=25\), so \(s^2=100/25=4\). Compare adjusted R-squared and \(C_p\).
Method. Calculate each criterion using the stated coefficient count, including the intercept, and then compare the directions of preference.
For the smaller model, residual mean square is \(120/(30-3)=120/27\). The total mean square is \(200/(30-1)=200/29\). Thus adjusted R-squared is
For the larger model, residual mean square is \(100/(30-5)=4\), giving
The \(C_p\) values are \(120/4-30+2(3)=30-30+6=6\) for the smaller model and \(100/4-30+2(5)=25-30+10=5\) for the larger model. Both adjusted R-squared and \(C_p\) favor the larger model in this comparison. This does not establish that it will have smaller error on future observations: the result relies on the model assumptions and on the reference variance estimate. A common mistake is to use \(n-q-1\) residual degrees of freedom after already counting the intercept in \(q\); that counts the intercept twice.
Check: The larger model’s \(RSS\) is lower, while its \(q\) is larger. The calculations show the fit improvement outweighs the adjustment under both criteria.
AIC and BIC Can Disagree About Prediction
For nested or non-nested Gaussian regression models fitted to the same response sample, compare changes in the criteria rather than their absolute values. If a larger model adds one mean coefficient, the change in AIC is \(n\log(RSS_{\text{large}}/RSS_{\text{small}})+2\), and the change in BIC is \(n\log(RSS_{\text{large}}/RSS_{\text{small}})+\log n\). A negative change favors the larger model; a positive change favors the smaller one.
Worked Example: Information Criteria Versus Held-Out Error
Two hypothetical regression models are fitted to the same \(n=40\) training observations. The smaller model has \(RSS=250\); the larger adds one coefficient and has \(RSS=230\). On a separate test set of 100 observations, their squared prediction errors sum to 280 and 260, respectively. Compare the AIC and BIC changes with test-set mean squared error.
Method. Use the larger-minus-smaller criterion changes, then divide each test-set squared-error sum by its 100 observations.
The training RSS ratio is \(230/250=0.92\). Therefore,
The negative change favors the larger model by AIC. For BIC,
The positive change favors the smaller model by BIC. On the test set, the smaller model’s mean squared error is \(280/100=2.80\), while the larger model’s is \(260/100=2.60\). The larger model has the lower observed test-set error. The rankings therefore differ: the test-set result agrees with AIC here, but not BIC.
Common mistake. Do not interpret BIC’s preference as proof that the smaller model has lower prediction error. BIC’s penalty is not a test-error estimate. Also, retain enough digits when adding the BIC terms: \(40\log(0.92)=-3.335264\ldots\) and \(\log(40)=3.688879\ldots\), giving \(0.3536\) to four decimals.
Check: The RSS ratio is below one, so the larger model improves training fit. Its AIC penalty is smaller than the fit gain, while its BIC penalty is larger. The test set is a separate comparison and happens to favor the larger model.
What a Test Set Adds—and What It Cannot Do
A properly held-out test set offers a direct estimate of the chosen model’s error under the test observations’ loss function. For squared-error loss, test mean squared error is the sum of squared prediction errors divided by the number of test observations. It is not automatically an unbiased estimate for every future setting: the test sample must represent the intended population, predictions must be made without using test outcomes, and the prediction task must be defined consistently.
A small test set can produce a noisy ranking. A few unusual observations may dominate squared error, and changing the random split can change which model appears better. Further, repeatedly comparing models against the same test set and adjusting them in response lets information from that set influence selection. The reported test error can then be optimistic. In that situation, an additional untouched evaluation sample is needed for a credible final assessment.
Worked Example: Calculating Test-Set Error
A hypothetical claims team holds out four observations to compare two models. Model A has prediction errors \(1,-2,3,0\); Model B has errors \(-2,1,2,-1\). Calculate each model’s test mean squared error and state the conclusion this small test set supports.
Method. Square each error, add the four squared values, and divide by four.
For Model A, the squared errors are \(1^2=1\), \((-2)^2=4\), \(3^2=9\), and \(0^2=0\). Their sum is \(1+4+9+0=14\), so its test mean squared error is \(14/4=3.50\).
For Model B, the squared errors are \((-2)^2=4\), \(1^2=1\), \(2^2=4\), and \((-1)^2=1\). Their sum is \(4+1+4+1=10\), so its test mean squared error is \(10/4=2.50\). Model B has the lower observed error on these four held-out cases.
Common mistake. A lower score on four observations does not establish that Model B has lower expected error across all future claims. The sample is small, and the conclusion applies directly only to this test sample and its chosen loss. Do not use these same test outcomes repeatedly to tune the model and still treat the final score as an untouched evaluation.
Check: The two squared-error totals are 14 and 10, and each is divided by four. The computed means are 3.50 and 2.50, respectively.
Why the Estimates Can Be Imperfect
Each method has a different source of uncertainty or potential mismatch:
- Model assumptions: The prediction interpretations of \(C_p\), AIC, and the Gaussian regression formulas depend on conditions such as an appropriate mean model and error distribution. Dependence, unequal variances, or misspecification can undermine a criterion’s intended interpretation.
- Variance estimation: \(C_p\) depends on the reference estimate \(s^2\). If that estimate is unstable or inappropriate for the candidate models, the resulting values can be misleading.
- Target mismatch: BIC’s model-evidence motivation differs from minimizing a particular future prediction loss. Adjusted R-squared is a degrees-of-freedom-adjusted fit measure, not a held-out score.
- Sampling variation: Both information criteria and test-set errors depend on the observed data. A different training sample or test split can change model rankings.
- Different loss functions: AIC in the Gaussian setting is linked to likelihood-based discrepancy, whereas a test set may be assessed by squared error or another chosen loss. Different targets need not rank models identically.
In practice, state what is being estimated before interpreting a ranking. AIC and \(C_p\) can help assess predictive performance under their assumptions; BIC and adjusted R-squared answer different questions. A genuinely independent test set measures realized error on its cases, but remains an estimate of future performance. Agreement among methods can be reassuring, while disagreement is a reason to examine their targets, assumptions, and data—not to declare one score universally correct.
Check Your Understanding
For each problem, use the stated convention that \(q\) counts mean-regression coefficients including the intercept.
- A regression has \(n=20\), \(q=3\), \(RSS=68\), and \(TSS=100\). Calculate adjusted R-squared. (Answer: \(1-(68/17)/(100/19)=0.2400\).)
- A candidate model has \(n=50\), \(q=4\), and \(RSS=180\). A reference model gives \(s^2=5\). Calculate \(C_p\) and say whether a negative value is mathematically possible. (Answer: \(180/5-50+2(4)=-6\); yes, the criterion can be negative.)
- With \(n=25\), a larger model adds one coefficient and reduces RSS to 0.90 times the smaller model’s RSS. Calculate the AIC change to four decimals and state the preferred model. (Answer: \(25\log(0.90)+2=-0.6340\); AIC favors the larger model.)
- With \(n=100\), a larger model adds one coefficient and has an RSS ratio of 0.98 relative to the smaller model. Calculate the BIC change to four decimals and state the preferred model. (Answer: \(100\log(0.98)+\log(100)=2.5849\); BIC favors the smaller model.)
- On a three-observation test set, Model A has errors \(1,-2,3\), and Model B has errors \(-2,1,0\). Calculate both test mean squared errors and state what can be concluded. (Answer: Model A has \(14/3=4.6667\), Model B has \(5/3=1.6667\); Model B has lower observed error on this test set, but the small set does not establish lower future expected error.)