Interpreting The Cp And Model-Size Calibration

Why Compare Cp With Model Size?

Mallows’ $C_p$ is useful not only as a score for comparing candidate regression models. Its relationship to model size provides a diagnostic: when a candidate model represents the mean response adequately, its $C_p$ will often be near the number of coefficients it estimates. A substantial excess can signal that the model has omitted important structure. This is a calibration guideline, not a guarantee that a particular model is correct.

The comparison is meaningful only when the coefficient count is defined consistently and the models use the same observations and error-variance estimate. Those details matter: an omitted intercept or a changing analysis sample can make an apparent calibration pattern misleading.

Definition: For a candidate linear regression model fitted to $n$ observations, let $RSS_p$ be its residual sum of squares and let $p$ be its number of fitted regression coefficients, including the intercept when one is fitted. Using an estimate $\widehat{\sigma}^2$ of the error variance, Mallows’ $C_p$ is $$ C_p=\frac{RSS_p}{\widehat{\sigma}^2}-n+2p. $$ The model-size calibration compares $C_p$ with $p$. The error-variance estimate is commonly obtained from a larger reference model; the implications of how it is estimated require separate care.

The familiar benchmark $C_p\approx p$ comes from an expectation under a correctly specified candidate model and a known error variance. In applied comparisons, the variance is estimated, so the benchmark is generally approximate. A value close to $p$ is evidence consistent with an adequate model, but it does not prove adequacy, and departures can arise from sampling variation.

Why the Calibration Holds

Consider a fixed-design linear regression setup. The response vector is $Y=\mu+\varepsilon$, where the mean vector $\mu$ is fixed and the errors have mean zero and covariance matrix $\sigma^2 I$. A candidate model has a design matrix of rank $p$, including the intercept if present. Its least-squares projection matrix is $H$, so its residual vector is $(I-H)Y$ and its residual sum of squares is $Y^\mathsf{T}(I-H)Y$.

If the candidate model contains the true mean vector, then $H\mu=\mu$, and therefore $(I-H)\mu=0$. Taking the expectation of the residual sum of squares gives

E[RSS_p] =E\!\left[\varepsilon^\mathsf{T}(I-H)\varepsilon\right] =\sigma^2\operatorname{tr}(I-H) =\sigma^2(n-p).

The last equality follows because $H$ is a rank-$p$ projection matrix, so $\operatorname{tr}(H)=p$ and $\operatorname{tr}(I-H)=n-p$. Substituting this expected residual sum of squares into the definition with the true variance $\sigma^2$ gives

E[C_p] =\frac{E[RSS_p]}{\sigma^2}-n+2p =(n-p)-n+2p=p.

This derivation uses the fixed-design, correctly specified model and a known variance in the calibration calculation. It does not say that every observed $C_p$ equals $p$. If the candidate model omits part of the true mean, then $(I-H)\mu\ne 0$, and

E[RSS_p]=(n-p)\sigma^2+\mu^\mathsf{T}(I-H)\mu.

The additional term is nonnegative: it is the squared length of the part of the mean that the candidate model cannot represent. With known $\sigma^2$, this makes the expected $C_p$ exceed $p$ by $\mu^\mathsf{T}(I-H)\mu/\sigma^2$. When $\sigma^2$ is estimated instead, the ratio in $C_p$ has additional variability, and the simple expectation calculation is not generally exact.

Reading a Candidate-Model Table

A useful interpretation considers both the absolute $C_p$ value and its distance from $p$. If $C_p$ is much larger than $p$, the candidate may leave substantial systematic signal unexplained. If it is near $p$, its fit and complexity are consistent with the calibration under the model assumptions. If $C_p$ is below $p$, that can occur from random variation; it is not proof that the model has discovered a uniquely optimal structure.

For models fitted to the same observations with a common $\widehat{\sigma}^2$, the change in $C_p$ makes the fit-complexity tradeoff explicit. If adding one coefficient reduces RSS by $\Delta RSS$, then

C_{p,\text{new}}-C_{p,\text{old}} =-\frac{\Delta RSS}{\widehat{\sigma}^2}+2.

Thus the added coefficient reduces $C_p$ when the RSS reduction is greater than $2\widehat{\sigma}^2$, leaves it unchanged when the reduction equals that threshold, and increases it when the reduction is smaller. This comparison concerns $C_p$; it is not by itself a test that the added coefficient is statistically significant.

Worked Example: Finding the Calibrated Region in a Model Table

A hypothetical portfolio study uses the same 40 policies for every candidate model. A reference model has $p_F=8$ fitted coefficients, including the intercept, and $RSS_F=128$. Its residual variance estimate is $\widehat{\sigma}^2=128/(40-8)=4$. The candidate models have the following residual sums of squares. Calculate $C_p$ and assess calibration.

Candidate coefficient count $p$	$RSS_p$
2	176
3	156
4	140
5	136

Method. Use $C_p=RSS_p/4-40+2p$, then compare each result with its own coefficient count.

For $p=2$, $176/4=44$, so $C_p=44-40+4=8$. This is 6 above the model size.

For $p=3$, $156/4=39$, so $C_p=39-40+6=5$. This is 2 above the model size.

For $p=4$, $140/4=35$, so $C_p=35-40+8=3$. This is 1 below the model size.

For $p=5$, $136/4=34$, so $C_p=34-40+10=4$. This is 1 below the model size.

The $p=2$ model is the least calibrated: its $C_p$ is well above its size, which is consistent with omitted signal. The $p=4$ and $p=5$ models are both near their respective sizes. The $p=4$ model has the smallest $C_p$, but the small differences should not be treated as proof that it is the true model.

Check. The candidate RSS values decrease as model size increases, as expected for nested least-squares fits. The calculations use $p$ as the total number of coefficients, not just the number of predictors. A common mistake is to call a low $C_p$ conclusive; the calibration is a diagnostic under assumptions, not a model-truth certificate.

Model Size and the Cost of an Extra Coefficient

The change formula helps explain why $C_p$ often settles near model size for models that have captured the important mean structure. Once the main signal has been represented, an added coefficient may reduce RSS only by the amount expected from fitting noise. The $2p$ term charges two units of scaled error for each fitted coefficient, balancing that fit improvement. A coefficient that barely reduces RSS may therefore raise $C_p$ rather than improve it.

Worked Example: An Added Predictor at the Cp Break-Even Point

Two nested models are fitted to the same $n=30$ hypothetical observations, using a common $\widehat{\sigma}^2=9$. The smaller model has $p=3$ coefficients and $RSS_3=252$. The larger model adds one coefficient and has $p=4$ and $RSS_4=234$. Calculate both $C_p$ values and interpret the change.

Smaller model. $252/9=28$, so $C_{p,3}=28-30+2(3)=28-30+6=4$.

Larger model. $234/9=26$, so $C_{p,4}=26-30+2(4)=26-30+8=4$.

The RSS reduction is $252-234=18$, while the break-even reduction is $2\widehat{\sigma}^2=2(9)=18$. The two models therefore have equal $C_p$. The larger model improves fit by exactly the amount needed to offset its one-coefficient penalty.

Check. The change formula gives $-18/9+2=0$, agreeing with the direct calculations. The trap is to treat every RSS reduction as an improvement in $C_p$: an added coefficient must reduce RSS by more than $2\widehat{\sigma}^2$ to lower this criterion.

Counting Coefficients and Interpreting Deviations

In a model with an intercept and three slopes, $p=4$, not $p=3$. The count is the rank of the fitted regression design matrix. This includes the intercept and all independently estimated slope coefficients; a categorical predictor represented by multiple indicator variables contributes the number of coefficients actually estimated. If a model is fit without an intercept, do not add one to its count.

A deviation $C_p-p$ can be informative because, with known variance, it corresponds in expectation to the candidate model's scaled lack of fit. In practice, both the residual sum of squares and the variance estimate are sample quantities. Therefore, interpret the size and direction of the deviation together with the candidate's purpose, the common-sample requirement, and the modeling assumptions.

Worked Example: Correcting an Intercept-Counting Error

A candidate model uses 28 hypothetical observations and has an intercept plus three slopes, so it estimates $p=4$ coefficients. Its RSS is 84, and the common variance estimate is $\widehat{\sigma}^2=3$. Calculate $C_p$, then show the error from counting only the three slopes.

Correct count. $84/3=28$. Thus $C_p=28-28+2(4)=8$. Compared with the correct model size $p=4$, this $C_p$ is 4 higher.

Incorrect count. If the intercept is omitted from the count, the calculation becomes $28-28+2(3)=6$. This is not the Mallows’ $C_p$ for the fitted model: it understates the score by 2 and compares it with the wrong model size.

Check. The model estimates four coefficients, so both occurrences of model size in the interpretation must use $p=4$. This example uses no mortality assumption, interest rate, or insurance payment model; it is a regression calculation.

Common Traps and Practical Use

Expecting exact equality. $C_p$ near $p$ is an expected calibration under stated conditions, not an identity for each sample.
Forgetting the intercept. Count all fitted coefficients, including the intercept when present.
Comparing incompatible fits. Use the same observations and a common variance estimate when interpreting a candidate-model table.
Calling $C_p<p$ proof of overfitting or optimality. Values on either side of $p$ can occur through sampling variation.
Ignoring the candidate model's assumptions. Heteroscedastic errors, dependence, or an incorrect mean structure can undermine the standard calibration.

In practice, the calibration is most useful as a reasonableness check alongside the candidate models' fit and complexity. A model with $C_p$ far above its size deserves scrutiny for missed structure; a model near its size is plausible under the assumptions but still requires judgment. This interpretation is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Key takeaway: Under a correctly specified fixed-design regression model with error variance $\sigma^2$, the expected residual sum of squares is $(n-p)\sigma^2$, giving expected $C_p=p$ when that variance is known. With an estimated variance, treat $C_p\approx p$ as a calibration guideline, not a guarantee.

Check Your Understanding

For each problem, use the stated coefficient count, including the intercept whenever the model has one.

A candidate model fitted to $n=25$ observations has $p=3$, $RSS_p=100$, and a common variance estimate of 4. Calculate $C_p$ and state whether it is above or below $p$. (Answer: $C_p=6$, which is 3 above $p$.)
Two nested models use the same data and variance estimate $\widehat{\sigma}^2=6$. Adding one coefficient reduces RSS from 210 to 198. By how much does $C_p$ change, and which model has the smaller value? (Answer: The change is $-12/6+2=0$; the models have equal $C_p$.)
For $n=36$ and $\widehat{\sigma}^2=3$, one candidate has $p=2$, $RSS_p=120$; another has $p=4$, $RSS_p=102$. Calculate both $C_p$ values and identify which is closer to its model size. (Answer: The values are 8 and 6, respectively; the $p=4$ model is closer to its size.)
A candidate model with an intercept and three slopes is fitted to $n=28$ observations. If $RSS_p=84$ and $\widehat{\sigma}^2=3$, what is $p$ and what is $C_p$? (Answer: $p=4$ and $C_p=8$.)
With known error variance $\sigma^2=2$, a candidate has $p=3$ and its squared lack-of-fit term $\mu^\mathsf{T}(I-H)\mu$ is 8. Under the fixed-design assumptions, what is its expected $C_p$? (Answer: $3+8/2=7$.)

What You'll Learn