Using Mallows Cp to Compare Models
A least-squares model can fit the observed data well simply by using enough predictors. The practical question is whether the improved fit is worth the added complexity. Mallows Cp gives a way to compare candidate models by combining their residual error with a penalty for the number of fitted regression coefficients. Its values are most informative when read alongside that coefficient count, rather than as a ranking on their own.
The calculations below use candidate models fitted to the same response values and the same observations. This matters: a difference in residual sum of squares is interpretable as a model difference only when the data being fitted are held fixed.
The shared variance estimate puts the candidate models’ residual sums of squares on a common scale. The term \(2p_m\) penalizes models for fitting more coefficients. Under the usual least-squares conditions—fixed predictor values, independent errors with mean zero and common variance, and a full model that provides an appropriate estimate of that variance—Cp serves as an estimate of prediction error, up to a common scale and additive constant. The formula and its prediction-error motivation were established in the earlier material on deriving Mallows Cp.
A useful reference is \(C_p\) close to \(p_m\). A large gap above \(p_m\) can signal that a candidate leaves substantial structure unexplained. A low Cp alone is not enough to select a model: the coefficient count and the assumptions behind the shared variance estimate matter too.
Worked Ranking for Several Candidates
Start by establishing one variance estimate for the entire candidate set. Then calculate each Cp with that same estimate. The following hypothetical example uses 30 observations and five candidate models, each fitted to the same response.
Worked Example: Ranking Five Candidate Regressions
Suppose the full model has \(p_F=5\) coefficients and \(RSS_F=250\) squared response units. Its residual degrees of freedom are \(30-5=25\), so the estimate used for every candidate is
The candidate models have the following residual sums of squares and coefficient counts. All RSS values are in squared response units.
| Candidate | \(p_m\) | \(RSS_m\) | Cp calculation | \(C_p\) |
|---|---|---|---|---|
| Intercept only | 1 | 900 | \(900/10-30+2(1)=90-30+2\) | 62 |
| Model A | 2 | 500 | \(500/10-30+2(2)=50-30+4\) | 24 |
| Model B | 3 | 330 | \(330/10-30+2(3)=33-30+6\) | 9 |
| Model C | 4 | 270 | \(270/10-30+2(4)=27-30+8\) | 5 |
| Full model | 5 | 250 | \(250/10-30+2(5)=25-30+10\) | 5 |
The intercept-only model has \(C_p=62\), far above its \(p_m=1\), and Model A also has a substantial gap: \(C_p=24\) compared with \(p_m=2\). Model B improves, but \(C_p=9\) remains notably above \(p_m=3\). Models C and the full model both have Cp of 5. Model C uses four coefficients rather than five, so it is a compact candidate with a value reasonably close to its coefficient count.
This is evidence in favor of considering Model C, not proof that it is the uniquely correct model. The full model’s Cp is also 5, and other aspects of the analysis—such as whether its assumptions are credible—still matter.
Why Cp Is Compared With the Coefficient Count
The reference \(C_p\approx p_m\) follows from the prediction-error rationale. If a candidate correctly represents the mean response and the error variance is known, its expected residual sum of squares is \((n-p_m)\sigma^2\). Substituting that expectation into the Cp formula gives
The approximation sign is appropriate in practical comparisons because \(\sigma^2\) is generally replaced by an estimate from the full model. If a candidate does not represent the mean response adequately, its residual error can include both random error and lack of fit. That extra error can push Cp well above the candidate’s coefficient count.
This reasoning also explains why Cp is not just a measure of fit or just a complexity score. \(RSS_m\) rewards better fit, while \(2p_m\) charges for the flexibility used to achieve it. A candidate with fewer coefficients may be attractive if it retains a low Cp and its Cp is near its own \(p_m\).
Worked Example: Comparing a Smaller Model With the Full Model
In a second hypothetical data set, \(n=24\). A full model with \(p_F=4\) coefficients has \(RSS_F=150\). A smaller candidate has \(p_m=3\) coefficients and \(RSS_m=170\). First estimate the common error variance from the full model:
For the three-coefficient candidate,
For comparison, the full model’s Cp is
The smaller candidate’s Cp, about 4.667, is reasonably close to its coefficient count of 3; the full model has Cp 4 and coefficient count 4. Both values are modest compared with, for example, the much larger Cp of a severely underfitting candidate. Cp alone does not settle whether the small increase in residual error justifies dropping a coefficient. The two models should be considered in context, including what was measured and whether the candidate set contains plausible alternatives.
A Useful Identity—and a Limitation
When the full model itself supplies \(\hat{\sigma}^2=RSS_F/(n-p_F)\), its own Cp has a simple algebraic form:
This equality follows from how the variance estimate was defined. It does not show that the full model is correctly specified, predicts well on new observations, or is preferable to a smaller model. In particular, the full model will always land exactly on the \(C_p=p\) reference under this calculation. It is not an independent confirmation of the full model’s adequacy.
Worked Example: Checking the Full-Model Identity
Suppose a hypothetical full model has \(n=18\), \(p_F=3\), and \(RSS_F=90\) squared response units. Then
Its Cp is
The result equals \(p_F=3\), as the identity requires. The calculation verifies the arithmetic and the reference value, but says nothing by itself about whether the full model is scientifically sensible or whether its variance estimate is trustworthy.
Limits and Common Mistakes
Mallows Cp is a comparison tool, not a guarantee of good prediction or a substitute for checking the model. Its usefulness depends on the candidate models and the variance estimate. If the full model omits important structure or otherwise provides an unsuitable estimate of the error variance, every candidate’s Cp can be distorted because the same denominator is used throughout. A poor candidate set creates a separate problem: the lowest or most promising Cp cannot point to a model that was never considered.
- Using a different variance estimate for each candidate. This changes the scale across models. Use a common estimate, typically from the designated full model.
- Choosing the smallest Cp without considering \(p_m\). Cp should be read together with the number of coefficients. Values near the \(C_p=p_m\) reference are useful evidence, not a mechanical pass-or-fail rule.
- Treating the full-model identity as validation. Its \(C_p=p_F\) follows algebraically from the full-model variance estimate. It is not an independent test of fit.
- Comparing models fitted to different observations or response scales. The residual sums of squares then do not provide a like-for-like comparison.
- Interpreting a favorable Cp as proof of causation or certainty. Cp does not establish causal effects, guarantee future performance, or remove uncertainty about which candidate model is best.
In practice, report the candidate’s Cp, coefficient count, and the variance estimate used. Then describe the ranking as evidence within the specified candidate set, and explain any assumptions that could affect it.
Check Your Understanding
Use the definitions and calculations above to answer these questions.
- A candidate model has \(n=20\), \(p_m=3\), \(RSS_m=180\), and the shared variance estimate is \(\hat{\sigma}^2=9\). Calculate its Cp and compare the result with \(p_m\).
- Why must candidate models use the same estimate of error variance when their Cp values are compared?
- Explain why the full model’s Cp equals its coefficient count when its own residual mean square supplies \(\hat{\sigma}^2\).
- A model has a lower Cp than a second model, but its Cp is far above its coefficient count. Why should the lower value not be used alone to declare it the better choice?
- Name one limitation that arises if the full model provides an unsuitable estimate of the error variance.