From Regression Output to Mallows’ \(C_p\)
Mallows’ \(C_p\) turns regression output into a score that combines fit and model size. Calculating it correctly requires more than substituting a residual sum of squares: the candidate models must use the same observations, the error-variance estimate must be shared across candidates, and the coefficient count must include the intercept when one is fitted.
This tutorial focuses on carrying out that calculation. The central practical task is often to extract four items from an output table: the sample size \(n\), a reference model’s residual sum of squares, its residual degrees of freedom, and each candidate model’s residual sum of squares and number of fitted coefficients.
Identify the Shared Variance Estimate
Suppose the reference model has \(p_{\mathrm{ref}}\) fitted coefficients and residual sum of squares \(\operatorname{RSS}_{\mathrm{ref}}\). If it is fitted to the same \(n\) observations as the candidate models, its residual degrees of freedom are \(n-p_{\mathrm{ref}}\). The usual unbiased error-variance estimate from this model is
Use that one estimate in the \(C_p\) calculation for every candidate. Do not estimate a different variance from each candidate’s own residual sum of squares: doing so changes the criterion and no longer makes the intended comparison.
The reference-model degrees of freedom depend on the number of fitted coefficients, not simply on the number of predictors. For a regression with an intercept and \(k\) slopes, \(p_{\mathrm{ref}}=k+1\), so the residual degrees of freedom are \(n-k-1\). If the model is fitted without an intercept, count only its fitted slopes.
Why the Reference Estimate Uses Residual Degrees of Freedom
The variance estimate divides by \(n-p_{\mathrm{ref}}\) because fitting \(p_{\mathrm{ref}}\) coefficients uses \(p_{\mathrm{ref}}\) degrees of freedom. One short justification follows from the geometry of least squares.
Assume the reference model is \(Y=X\beta+\varepsilon\), where \(X\) is a fixed, full-column-rank design matrix with \(p_{\mathrm{ref}}\) columns, and the errors are independent with mean zero and common variance \(\sigma^2\). Let \(H=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}\) be the projection matrix onto the columns of \(X\), and let \(M=I-H\). The residual vector is \(M Y=M\varepsilon\), since \(MX=0\). Thus the residual sum of squares is \(\varepsilon^\mathsf{T}M\varepsilon\). Its expected value is
The last equality holds because \(M\) is an idempotent projection matrix of rank \(n-p_{\mathrm{ref}}\), so its trace is that rank. Dividing the reference residual sum of squares by \(n-p_{\mathrm{ref}}\) therefore gives an unbiased estimate of \(\sigma^2\). This is the variance estimate inserted into \(C_p\).
A Reliable Calculation Sequence
Keep the reference-model and candidate-model counts separate. A compact calculation table can prevent the most common arithmetic and bookkeeping errors.
Check that the reference and candidate residual sums of squares come from the same response observations, so the same \(n\) applies.
Include an intercept if fitted, then calculate \(n-p_{\mathrm{ref}}\).
Divide \(\operatorname{RSS}_{\mathrm{ref}}\) by \(n-p_{\mathrm{ref}}\).
For every model, count all fitted slopes and its intercept, if present.
Calculate \(\operatorname{RSS}_p/\widehat{\sigma}^{\,2}-n+2p\), keeping the shared estimate unchanged across candidates.
Worked Example: Two Candidate Models and a Shared Estimate
A reference regression uses \(n=25\) observations, has six fitted coefficients, and has \(\operatorname{RSS}_{\mathrm{ref}}=120\). Two candidate models use the same observations. The first has two slopes and \(\operatorname{RSS}=150\); the second has four slopes and \(\operatorname{RSS}=108\). Calculate the common variance estimate and both \(C_p\) values. The models include intercepts.
Method and coefficient counts. The reference model has \(p_{\mathrm{ref}}=6\), so its residual degrees of freedom are \(25-6=19\). Each candidate also has an intercept: the first has \(p=2+1=3\), and the second has \(p=4+1=5\).
Common variance estimate.
For the first candidate, use its residual sum of squares and \(p=3\):
For the second candidate, use \(p=5\) and the same estimate:
Check. The variance estimate uses the reference model’s 19 residual degrees of freedom, not \(25\). Both candidate calculations use the same \(120/19\). The results are \(C_p=4.75\) and \(C_p=2.1\). The common mistake to avoid is replacing \(120/19\) by \(120/25\), which incorrectly treats the reference residual degrees of freedom as the sample size.
Worked Example: Reading Counts From a Regression Summary
A hypothetical severity study fits a reference regression with an intercept and six slopes to \(40\) policy-level observations. Its residual sum of squares is \(264\). Two candidate models, fitted to the same observations with intercepts, have the following output: Candidate A has three slopes and \(\operatorname{RSS}=320\); Candidate B has five slopes and \(\operatorname{RSS}=280\). Calculate the shared variance estimate and each candidate’s \(C_p\).
Method and reference degrees of freedom. The reference model has \(1+6=7\) coefficients. Its residual degrees of freedom are \(40-7=33\), so
Candidate A. Its three slopes and intercept give \(p_A=4\). Therefore,
Candidate B. It has \(p_B=5+1=6\) coefficients, so
Check. The reference residual degrees of freedom are \(33\), and both candidate scores use variance estimate \(8\). Candidate B’s calculation uses six coefficients, not five, because the intercept is fitted. Omitting that intercept would incorrectly reduce its complexity term by \(2\).
Worked Example: Checking a Candidate Against the Reference Model
A reference model with an intercept and four slopes is fitted to \(30\) observations, with \(\operatorname{RSS}_{\mathrm{ref}}=100\). Two smaller models use the same observations and include intercepts. Model A has one slope and \(\operatorname{RSS}=140\); Model B has three slopes and \(\operatorname{RSS}=112\). Calculate all three \(C_p\) values, including the reference model’s.
Method and common variance estimate. The reference model has \(p_{\mathrm{ref}}=5\), giving \(30-5=25\) residual degrees of freedom. Thus
Model A. Its intercept and one slope give \(p_A=2\):
Model B. Its intercept and three slopes give \(p_B=4\):
Reference-model check. For the reference model, \(p_{\mathrm{ref}}=5\) and \(\operatorname{RSS}_{\mathrm{ref}}/\widehat{\sigma}^{\,2}=100/4=25\). Hence
Check. The reference model’s value is \(5\), equal to its coefficient count. This follows algebraically when its own residual-based variance estimate is used: \(\operatorname{RSS}_{\mathrm{ref}}/\widehat{\sigma}^{\,2}=n-p_{\mathrm{ref}}\), so \(C_{p,\mathrm{ref}}=(n-p_{\mathrm{ref}})-n+2p_{\mathrm{ref}}=p_{\mathrm{ref}}\). It is a useful arithmetic check, not a substitute for counting coefficients correctly.
Common Calculation Traps
Most incorrect \(C_p\) calculations are caused by mixing quantities that look similar in a regression table. Check the definitions before substituting.
- Dividing by \(n\) instead of reference residual degrees of freedom. Use \(n-p_{\mathrm{ref}}\) for the reference variance estimate. In the first worked example, that denominator is \(19\), not \(25\).
- Forgetting the intercept. A model with two slopes and an intercept has \(p=3\). The \(2p\) term is then \(6\), not \(4\).
- Using a separate variance estimate for each candidate. The criterion uses the common reference estimate for every candidate in the comparison.
- Mixing sample sizes. If models were fitted to different observations because of missing values, their raw residual sums of squares are not directly comparable in this calculation. Verify a common analysis sample first.
- Counting the variance as a regression coefficient. The \(p\) in this formula counts fitted regression coefficients; it does not include \(\sigma^2\).
A useful final audit is to write down the inputs before calculating: \(n\), \(\operatorname{RSS}_{\mathrm{ref}}\), \(p_{\mathrm{ref}}\), \(\widehat{\sigma}^{\,2}\), and then each candidate’s \(\operatorname{RSS}_p\) and \(p\). This makes it easier to spot a mismatched sample, a mistaken denominator, or an omitted intercept.
Accurate calculation is essential before comparing candidate scores. The formula is simple, but the inputs must have consistent meanings: the reference model supplies the common variance estimate, while each candidate supplies its own fit and coefficient count.
Check Your Understanding
For each problem, use the same common-variance procedure and count the intercept whenever the statement says it is fitted.
- A reference model fitted to \(18\) observations has \(4\) coefficients and \(\operatorname{RSS}_{\mathrm{ref}}=56\). A candidate with an intercept and one slope has \(\operatorname{RSS}=70\). Find the variance estimate and the candidate’s \(C_p\). (Answer: \(\widehat{\sigma}^{\,2}=4\); \(C_p=3.5\))
- A reference model has \(n=32\), six coefficients, and \(\operatorname{RSS}_{\mathrm{ref}}=156\). A candidate with an intercept and two slopes has \(\operatorname{RSS}=210\). Find the candidate’s \(C_p\). (Answer: \(\widehat{\sigma}^{\,2}=6\); \(C_p=9\))
- A reference model has \(n=50\), eight coefficients, and \(\operatorname{RSS}_{\mathrm{ref}}=336\). A candidate with an intercept and four slopes has \(\operatorname{RSS}=400\). Find the variance estimate and \(C_p\). (Answer: \(\widehat{\sigma}^{\,2}=8\); \(C_p=10\))
- A reference model with an intercept and four slopes has \(\operatorname{RSS}_{\mathrm{ref}}=95\) on \(24\) observations. Two candidates, both with intercepts, have respectively one slope and \(\operatorname{RSS}=120\), and three slopes and \(\operatorname{RSS}=100\). Calculate both \(C_p\) values. (Answer: \(\widehat{\sigma}^{\,2}=5\); both candidates have \(C_p=4\))
- A reference model fitted to \(60\) observations has nine coefficients and \(\operatorname{RSS}_{\mathrm{ref}}=408\). Two candidates with intercepts have respectively two slopes and \(\operatorname{RSS}=480\), and five slopes and \(\operatorname{RSS}=432\). Calculate their \(C_p\) values using one shared estimate. (Answer: \(\widehat{\sigma}^{\,2}=8\); both candidates have \(C_p=6\))