How The Error-Variance Estimate Affects Cp

Why the Error-Variance Estimate Matters

Mallows’ $C_p$ balances a candidate model’s residual sum of squares against a penalty for the number of fitted parameters. Its calculation also requires an estimate of the regression error variance. Changing that estimate can shift every candidate’s $C_p$, and it can sometimes reverse the ranking of models. The key is to distinguish a change in the numerical scale of the criterion from a change in which model has the smaller value.

In a comparison, use the same variance estimate for every candidate. A common choice is the residual mean square from a sufficiently comprehensive full model. An estimate from an independent source may also be used when it is appropriate for the same error process. The estimate should not be chosen separately to suit each candidate.

Definition: For a candidate linear regression model fitted to $n$ observations, let $\operatorname{RSS}_j$ be its residual sum of squares and let $p_j$ be its number of fitted regression coefficients, including the intercept when present. Given one common estimate $\widehat{\sigma}^2$ of the error variance, Mallows’ criterion is $$ C_p(j)=\frac{\operatorname{RSS}_j}{\widehat{\sigma}^2}-n+2p_j. $$

The estimate $\widehat{\sigma}^2$ must be positive and expressed in the same squared response units as the residual sum of squares. The parameter count $p_j$ is not just the number of predictors when the model also has an intercept. For example, an intercept and two predictors give $p_j=3$.

How a Common Estimate Affects Comparisons

For two candidate models $A$ and $B$, subtract their criteria. The sample-size term cancels:

C_p(B)-C_p(A) = \frac{\operatorname{RSS}_B-\operatorname{RSS}_A}{\widehat{\sigma}^2} +2(p_B-p_A).

This expression explains why the variance estimate can affect the ranking. The estimate scales the difference in residual sums of squares, while the difference in parameter counts contributes a fixed penalty. If $B$ uses more parameters and reduces RSS, a smaller variance estimate gives that reduction more weight. A larger variance estimate gives it less weight relative to the extra-parameter penalty.

If two models have the same number of parameters, the penalty difference is zero. Their ranking is then determined by their RSS values alone, regardless of the positive common variance estimate. With different parameter counts, a ranking reversal is possible.

Worked Comparisons

Worked Example: Correcting the Scale of Two Cp Values

Suppose $n=18$. Candidate $A$ has $\operatorname{RSS}_A=44$ and $p_A=2$; candidate $B$ has $\operatorname{RSS}_B=39$ and $p_B=3$. First use the common variance estimate $\widehat{\sigma}^2=2$. Which model has the smaller $C_p$?

Method. Substitute the same estimate, sample size, and each model’s parameter count into the definition. The common mistake to avoid is miscalculating the RSS term before applying the sample-size and complexity terms.

For $A$, the scaled RSS is $44/2=22$, and the penalty is $2p_A=4$. Thus $$ C_p(A)=22-18+4=8.000. $$

For $B$, the scaled RSS is $39/2=19.5$, and the penalty is $2p_B=6$. Thus $$ C_p(B)=19.5-18+6=7.500. $$

Therefore $B$ has the smaller value by $8.000-7.500=0.500$. As a check, the difference formula gives $$ C_p(B)-C_p(A)=\frac{39-44}{2}+2(3-2)=-2.5+2=-0.5, $$ which agrees with the directly calculated values.

Worked Example: A Ranking Reversal as the Estimate Changes

Use the same candidate data: $n=18$, $\operatorname{RSS}_A=44$, $p_A=2$, $\operatorname{RSS}_B=39$, and $p_B=3$. Now suppose the common estimate is $\widehat{\sigma}^2=4$. Recalculate both criteria and compare the result with the preceding calculation.

Method. Recompute each criterion using the new common denominator. The common mistake is to assume that because $B$ had the smaller value before, its ranking cannot change.

For $A$, $44/4=11$, so $$ C_p(A)=11-18+2(2)=11-18+4=-3.000. $$

For $B$, $39/4=9.75$, so $$ C_p(B)=9.75-18+2(3)=9.75-18+6=-2.250. $$

Now $A$ has the smaller value, since $-3.000 < -2.250$. The ranking has reversed. The difference formula checks the result: $$ C_p(B)-C_p(A)=\frac{39-44}{4}+2(3-2)=-1.25+2=0.75. $$

Both criteria are negative under this plug-in estimate. A negative value is not, by itself, an arithmetic error: $C_p$ includes the subtraction of $n$. Its interpretation and calibration depend on the variance estimate being appropriate for the errors. The ranking calculation is still exactly as shown.

Result: For candidates $A$ and $B$, with $p_B>p_A$, a tie occurs when $$ \widehat{\sigma}^2 = \frac{\operatorname{RSS}_A-\operatorname{RSS}_B}{2(p_B-p_A)}, $$ provided $\operatorname{RSS}_A>\operatorname{RSS}_B$. At estimates below this threshold, the RSS improvement can outweigh the extra-parameter penalty; above it, the penalty can dominate.

Worked Example: Finding the Tie Threshold

Suppose candidate $A$ has RSS $50$ and $p_A=2$, while candidate $B$ has RSS $42$ and $p_B=4$, on the same $n=20$ observations. Find the variance estimate at which their $C_p$ values tie, then compare their rankings at estimates of $1$ and $4$.

Method. First equate the difference formula to zero to find the threshold. Then calculate or compare the criteria on either side. The common mistake is to compare RSS values alone, ignoring that $B$ has two additional parameters.

The difference is $$ C_p(B)-C_p(A)=\frac{42-50}{\widehat{\sigma}^2}+2(4-2) =-\frac{8}{\widehat{\sigma}^2}+4. $$

Setting this equal to zero gives $8/\widehat{\sigma}^2=4$, so the tie occurs at $\widehat{\sigma}^2=2$. At an estimate of $1$, the difference is $-8+4=-4$, so $B$ is lower. Directly, $C_p(A)=50-20+4=34$ and $C_p(B)=42-20+8=30$.

At an estimate of $4$, the difference is $-2+4=2$, so $A$ is lower. Direct calculation gives $$ C_p(A)=\frac{50}{4}-20+4=12.5-16=-3.5, $$ $$ C_p(B)=\frac{42}{4}-20+8=10.5-12=-1.5. $$

The difference is $(-1.5)-(-3.5)=2$, confirming the comparison. The lower RSS for $B$ is not enough to make it preferable under every variance estimate; its larger parameter count matters.

Where the Variance Estimate Comes From

A common choice is the residual mean square from a full model. If that model has $q$ fitted regression coefficients, including the intercept, and is fitted by least squares under the usual linear regression assumptions, its residual variance estimate is

\widehat{\sigma}^2_{\text{full}} = \frac{\operatorname{RSS}_{\text{full}}}{n-q}.

This estimate uses the full model’s residual degrees of freedom. It is not generally the residual mean square of each candidate being evaluated. Using one reference estimate makes the candidate criteria comparable on a common scale.

Worked Example: Using the Full-Model Residual Mean Square

There are $n=24$ observations. A full model has $q=6$ coefficients and residual sum of squares $36$. Two candidate models have $(\operatorname{RSS}_A,p_A)=(60,2)$ and $(\operatorname{RSS}_B,p_B)=(50,4)$. Use the full-model residual mean square for both candidates and identify the lower-$C_p$ model.

Method. Calculate the full-model variance estimate first, then use it unchanged in each candidate’s criterion. The common mistake is to use a candidate’s own RSS and residual degrees of freedom as its denominator.

The full model has $n-q=24-6=18$ residual degrees of freedom. Therefore $$ \widehat{\sigma}^2_{\text{full}}=\frac{36}{18}=2. $$

For $A$, $60/2=30$, and $2p_A=4$, giving $$ C_p(A)=30-24+4=10.000. $$

For $B$, $50/2=25$, and $2p_B=8$, giving $$ C_p(B)=25-24+8=9.000. $$

Thus $B$ is lower by $1.000$. Check using the difference formula: $$ C_p(B)-C_p(A)=\frac{50-60}{2}+2(4-2)=-5+4=-1. $$

The calculation is conditional on using the full-model estimate $2$ as the common variance estimate. If an appropriate external estimate were $4$, the same candidate data would instead give $C_p(A)=60/4-24+4=-5.000$ and $C_p(B)=50/4-24+8=-3.500$, so $A$ would be lower. This illustrates why the estimate’s source and suitability should be stated.

Why the Formula Has This Sensitivity

The effect of the estimate follows directly from subtracting the two definitions; no separate model-fitting argument is needed. Both models are evaluated on the same observations, so both contain the same $-n$ term. Subtraction removes it, leaving the scaled RSS difference and the difference in complexity penalties:

Write each criterion.
$C_p(A)=\operatorname{RSS}_A/\widehat{\sigma}^2-n+2p_A$ and $C_p(B)=\operatorname{RSS}_B/\widehat{\sigma}^2-n+2p_B$, using one common estimate.

Subtract $A$ from $B$.
The terms $-n$ cancel, and the common denominator allows the RSS terms to combine into $(\operatorname{RSS}_B-\operatorname{RSS}_A)/\widehat{\sigma}^2$.

Interpret the two remaining terms.
The scaled RSS difference rewards improved fit; $2(p_B-p_A)$ charges for extra parameters. Changing the estimate changes the relative weight of the fit difference, so a tie or reversal can result.

This comparison assumes both candidates use the same response and analysis sample, so their RSS values and $n$ refer to the same observations. If samples differ, the direct comparison is not valid in this form. The linear regression assumptions and the suitability of the variance estimate matter for the usual prediction-error interpretation of $C_p$; a calculation alone cannot establish that the estimate is appropriate.

Common Mistakes and Practical Checks

Arithmetic slips in the scaled RSS. With $n=18$, $\operatorname{RSS}=44$, $p=2$, and $\widehat{\sigma}^2=2$, the correct calculation is $44/2-18+2(2)=22-18+4=8.000$, not $10.000$. For RSS $39$ and $p=3$, it is $39/2-18+2(3)=19.5-18+6=7.500$, not $9.500$.
Changing the denominator across candidates. A separate residual variance estimate for each candidate changes the scale of the fit term from one model to another. It no longer gives the standard common-estimate $C_p$ comparison.
Ignoring the parameter count. Include the intercept if the model has one. A model with three predictors and an intercept has $p=4$.
Assuming the ranking cannot change. When models have different parameter counts, check the difference formula or the tie threshold before concluding.
Interpreting a negative value as automatic failure. The sign alone does not decide whether the computation is wrong. Verify the inputs, then assess whether the chosen variance estimate is suitable.

For a practical comparison, record the variance estimate and its source, confirm that all candidates use the same sample, count every fitted coefficient consistently, and verify the result by calculating $C_p(B)-C_p(A)$. This provides a quick check on both arithmetic and ranking.

Check Your Understanding

Use one common variance estimate for candidates in each comparison unless the question explicitly asks you to examine an unsuitable candidate-specific calculation.

For $n=16$ and $\widehat{\sigma}^2=1.5$, candidate $A$ has RSS $30$ and $p_A=2$; candidate $B$ has RSS $27$ and $p_B=3$. Calculate both values and state the comparison. (Answer: $C_p(A)=8.000$, $C_p(B)=8.000$; tie.)
For candidates $A$ and $B$, let $\operatorname{RSS}_A=90$, $p_A=2$, $\operatorname{RSS}_B=78$, and $p_B=5$, on the same sample. Find the tie value of $\widehat{\sigma}^2$, and identify the lower-$C_p$ candidate at estimates $1$ and $3$. (Answer: Tie at $\widehat{\sigma}^2=2$; $B$ is lower at $1$, and $A$ is lower at $3$.)
A full model fitted to $28$ observations has $q=7$ coefficients and RSS $42$. Candidate $A$ has RSS $56$ and $p_A=2$; candidate $B$ has RSS $50$ and $p_B=4$. Using the full-model residual mean square, calculate both criteria and identify the lower one. (Answer: $\widehat{\sigma}^2=42/(28-7)=2$; $C_p(A)=4.000$, $C_p(B)=5.000$; $A$ is lower.)
For $n=20$, candidate $A$ has RSS $40$, $p_A=2$, and candidate $B$ has RSS $35$, $p_B=3$. Using a common estimate $\widehat{\sigma}^2=2$, calculate both criteria. Why should you not replace the common estimate with a separate residual mean square for each candidate? (Answer: $C_p(A)=4.000$, $C_p(B)=3.500$; $B$ is lower. Candidate-specific denominators change the scale across models and do not give the standard common-estimate comparison.)

What You'll Learn