Why Mallows \(C_p\) Is Used
A least-squares model with more predictors can usually reduce the residual sum of squares, but a smaller residual sum of squares alone does not establish that the model is preferable. Mallows \(C_p\) puts residual fit and model size on a shared scale: it compares a candidate model’s residual sum of squares with an estimate of error variance, then applies a penalty for the number of regression parameters.
The criterion is useful when comparing candidate least-squares models fitted to the same response data. Its value is not a p-value, a test statistic for a hypothesis, or a direct probability that a model is correct. It is a model-selection measure whose interpretation depends on how the candidate models and the variance estimate were formed.
Definition and Working Assumptions
Suppose there are \(n\) observations on a quantitative response \(Y\). The candidate models are fitted by least squares to the same response values and observations. For the standard interpretation of \(C_p\), use a larger reference model that includes the candidate’s predictors and is considered adequate for representing the mean response. A common working model is that the errors are independent, have mean zero, and have a common variance \(\sigma^2\). A normal error model is often used for least-squares modelling, but the \(C_p\) calculation itself uses residual sums of squares and a variance estimate.
Let \(RSS_c\) be the residual sum of squares from a candidate model, and let \(p_c\) be its number of estimated regression parameters. For the usual full-rank regression with an intercept, \(p_c\) counts the intercept and all slope coefficients. Let the larger reference model have \(p_{\mathrm{ref}}\) estimated regression parameters and residual sum of squares \(RSS_{\mathrm{ref}}\). Its residual mean square provides the variance estimate \(\widehat{\sigma}^2=RSS_{\mathrm{ref}}/(n-p_{\mathrm{ref}})\).
The subtraction of \(n\) is part of the criterion, not an additional parameter adjustment. The candidate and reference models must use the same response observations, and the residual sum of squares and variance estimate must be in the same response units. These conditions make the scaled fit term interpretable and allow the candidate models to be compared fairly.
What the Formula Balances
The first term, \(RSS_c/\widehat{\sigma}^2\), measures the candidate model’s residual variation relative to the estimated error variance. For a fixed variance estimate and data set, a smaller candidate \(RSS_c\) reduces \(C_p\). But adding parameters generally improves least-squares fit, so the formula also includes the complexity penalty \(2p_c\). The \(-n\) term centers the criterion relative to the size of the data set.
The balance can be made explicit by rearranging the definition:
This is an algebraic rewriting of the same definition: the scaled residual fit is offset by \(n\), and the candidate’s parameter count adds a penalty. The criterion therefore does not simply reward the model with the smallest residual sum of squares. A more complex candidate must reduce residual variation enough to compensate for its larger penalty.
A useful reference point is \(C_p\) near \(p_c\). In standard applications, a candidate that represents the mean response adequately often has a \(C_p\) value of roughly this size. This is a practical benchmark, not an identity that every correct model must satisfy. A value considerably above \(p_c\) can indicate that the candidate’s residual variation is large relative to the reference estimate. A value below \(p_c\) is possible and does not, by itself, prove that the model is wrong or correct.
Worked Example: Calculating \(C_p\) for One Candidate
Suppose a hypothetical data set contains \(n=24\) observations. A reference model has an intercept and three slopes, so \(p_{\mathrm{ref}}=4\), and its residual sum of squares is \(40\). A candidate model has an intercept and one slope, so \(p_c=2\), with \(RSS_c=50\). Assume both least-squares fits use the same observations, the reference model is adequate for estimating the common error variance, and the error assumptions described above are reasonable.
First calculate the reference model’s residual degrees of freedom and variance estimate:
Substitute the candidate values into the definition:
The candidate has \(p_c=2\), so its \(C_p\) is 3 units above the \(p_c\) benchmark. This calculation alone does not determine whether the model should be selected; it says how the candidate’s residual fit and parameter penalty combine under the chosen variance estimate.
Using the Reference Model’s Variance Estimate
The variance estimate in \(C_p\) is not normally calculated from the candidate model being assessed. Using the reference model’s residual mean square gives the candidates a shared estimate of the noise level against which to judge their residual sums of squares. If each candidate supplied its own variance estimate in the formula, the criterion would no longer have the stated form or provide this common scale.
The reference model must be larger than the candidates in the relevant sense and should be adequate for the mean structure. If important predictors are omitted from the reference model, its residual variation may reflect both random error and a poorly represented mean. Then its residual mean square may not serve as a suitable estimate of the error variance, and the resulting \(C_p\) values may be misleading.
Worked Example: Computing the Variance Estimate First
In a hypothetical analysis, \(n=30\) observations are fitted with a reference model containing an intercept and four slopes. Its residual sum of squares is \(75\). A candidate with an intercept and two slopes has \(RSS_c=90\). Assume the same 30 observations are used for both fits, the reference model is adequate, and independent equal-variance errors are a reasonable working assumption.
The reference model has \(p_{\mathrm{ref}}=5\), giving \(30-5=25\) residual degrees of freedom. Therefore,
The candidate has \(p_c=3\), including its intercept. Its \(C_p\) is
The candidate’s \(C_p\) is 3 units above its parameter count. The number 6 is a criterion value, not an estimate of the variance: the variance estimate here is \(3\), in squared response units. Keeping those quantities distinct matters when reporting and interpreting the calculation.
Comparing the Fit-Complexity Balance
When candidate models use the same observations and reference variance estimate, their \(C_p\) values can be compared directly. A smaller value indicates a more favorable balance according to this criterion. The comparison should still be interpreted alongside the model’s purpose and the plausibility of its specification. \(C_p\) does not establish causation, confirm that assumptions hold, or replace subject-matter reasoning about which predictors belong in the analysis.
Two candidates can have equal \(C_p\) even when one has a smaller residual sum of squares. In that case, the smaller residual sum of squares may be offset by the larger parameter penalty. This illustrates why the criterion is about a balance rather than a ranking by fit alone.
Worked Example: Equal \(C_p\) Values From Different Models
Consider a hypothetical data set with \(n=18\) observations. A reference model has an intercept and three slopes, so \(p_{\mathrm{ref}}=4\), and \(RSS_{\mathrm{ref}}=28\). Two candidates use the same observations: candidate A has an intercept and one slope, with residual sum of squares 34; candidate B has an intercept and two slopes, with residual sum of squares 30. Suppose the reference model is adequate and the independent, equal-variance error assumption is reasonable.
The shared variance estimate is
For candidate A, \(p_A=2\), so
For candidate B, \(p_B=3\), so
Both candidates have \(C_p=3\). Candidate B has a smaller residual sum of squares, but it also has one more regression parameter. Under this criterion, those differences balance exactly in this example. Equal criterion values do not mean the fitted models are identical; they mean that the \(C_p\) calculation gives them the same value.
Common Mistakes and Practical Checks
- Using the candidate’s residual mean square as \(\widehat{\sigma}^2\). The stated \(C_p\) uses the variance estimate from the larger reference model, shared across candidates.
- Leaving out the intercept from the parameter count. In an ordinary full-rank regression with an intercept, count that intercept along with the slopes.
- Comparing models fitted to different observations. Different response samples can have different residual scales, so the resulting \(C_p\) values are not a fair direct comparison.
- Treating \(C_p\) as a hypothesis test or a hard cutoff. The value is a criterion, not a p-value, and being above or below \(p_c\) is not a formal pass-or-fail decision.
- Choosing the smallest residual sum of squares without considering complexity. Least-squares fit usually improves as predictors are added; \(C_p\) includes a penalty precisely so fit is not the only consideration.
A practical calculation check is to write down, before substituting, the common sample size \(n\), the reference model’s \(RSS_{\mathrm{ref}}\) and \(p_{\mathrm{ref}}\), and the candidate’s \(RSS_c\) and \(p_c\). Calculate the variance estimate separately, verify that it is in squared response units, and then apply the formula. Keeping those quantities explicit reduces the risk of mixing up the two models.
Check Your Understanding
Use the definition and assumptions in this tutorial to answer the following questions.
- A candidate has an intercept and three slopes. What is \(p_c\) for the usual full-rank model?
- A reference model has \(n=40\), \(p_{\mathrm{ref}}=6\), and \(RSS_{\mathrm{ref}}=68\). Calculate \(\widehat{\sigma}^2\).
- Why should candidate models being compared use the same observations and the same reference variance estimate?
- What does it mean, as a practical benchmark, for \(C_p\) to be near \(p_c\)? Why is that not a formal proof that the model is correct?
- A candidate has a smaller residual sum of squares than another candidate but a larger \(C_p\). How can the parameter penalty account for this result?