Deriving Mallows Cp as a Prediction-Error Estimate

Why Cp Has a Fit Term and a Penalty

A regression model with more predictors can fit the observed responses at least as well as a smaller model, because adding predictors cannot increase the least-squares residual sum of squares. But a smaller residual sum of squares alone does not show that the model will estimate the underlying mean responses well: fitting extra noise can make its fitted values less reliable. Mallows’ \(C_p\) combines a measure of fit with a penalty for the number of fitted parameters.

The rationale can be derived by asking a precise question: across repeated samples with the same predictor values, what is the expected squared distance between a model’s fitted mean responses and the true mean responses? The derivation explains both the form of \(C_p\) and why it needs an estimate of the error variance from a fuller model.

Setup and Prediction-Error Target

Suppose there are \(n\) observations with fixed predictor values. Write the response vector as \(y=\mu+\varepsilon\), where \(\mu\) is the vector of true mean responses at those predictor values and \(\varepsilon\) is the vector of errors. Assume the errors have mean zero, common variance \(\sigma^2\), and are independent. Normality is not needed for the expectation calculation below. Assume each candidate regression model is fit by least squares, has \(k\) identifiable parameters, and includes an intercept when appropriate.

For a candidate model with design matrix \(X_m\), let \(H_m=X_m(X_m^\mathsf{T}X_m)^{-1}X_m^\mathsf{T}\) be its projection matrix, assuming the columns are linearly independent. The fitted response vector is \(\hat{\mu}_m=H_my\). Its residual sum of squares is \(RSS_m=\|y-\hat{\mu}_m\|^2\). The parameter count \(k\) includes the intercept if the model has one.

The model need not represent the true mean vector exactly. Define its lack-of-fit vector as \(\delta_m=(I-H_m)\mu\). The length squared \(\|\delta_m\|^2\) measures the part of the true mean vector that the candidate model cannot represent. It is zero when \(\mu\) lies in the model’s column space.

Deriving the Prediction-Error Identity

The least-squares fitted means are \(H_my=H_m\mu+H_m\varepsilon\). Subtracting the true means gives \(H_m\mu-\mu+H_m\varepsilon=-\delta_m+H_m\varepsilon\). Squaring and taking expectations separates systematic lack of fit from random estimation error: the cross-product has expectation zero because \(E(\varepsilon)=0\).

For the third line, \(H_m\) is a symmetric projection matrix, so \(H_m^\mathsf{T}H_m=H_m\). The error covariance matrix is \(\sigma^2I\), which gives \(E(\varepsilon^\mathsf{T}H_m\varepsilon)=\sigma^2\operatorname{tr}(H_m)\). A projection onto a \(k\)-dimensional model space has trace \(k\). Thus the expected squared error consists of squared lack of fit plus a variance contribution of \(k\sigma^2\).

The residuals are \((I-H_m)y\). Since \((I-H_m)H_m=0\), their systematic component is \(\delta_m\); their random component lies in the residual space, whose dimension is \(n-k\). Therefore their expected squared length is

Subtracting this expected residual sum of squares from the expected prediction error shows what correction is needed to turn fit into an estimate of prediction error:

Worked Examples of the Derivation

Why the Variance Estimate Comes from a Fuller Model

The identity uses the common error variance \(\sigma^2\), which is usually unknown. A fuller model is used to estimate it because, if that model contains the true mean vector and has \(k_F\) parameters, its lack-of-fit vector is zero. Its residual sum of squares then has expectation \((n-k_F)\sigma^2\), so

is an unbiased estimator of \(\sigma^2\). Using this estimate in place of \(\sigma^2\) produces the commonly reported plug-in criterion \(\widehat{C}_{p,m}=RSS_m/\hat{\sigma}^2_F-n+2k\). Because the variance estimate is itself random, this substitution does not make the resulting ratio an exact unbiased estimator in every finite sample. The derivation gives the exact expectation rationale when the variance is known; the practical criterion uses the fuller-model estimate to calibrate it.

A fuller model does not automatically provide a good variance estimate. If even that model misses part of the true mean structure, then \(E(RSS_F)=\|\delta_F\|^2+(n-k_F)\sigma^2\). Dividing by its residual degrees of freedom includes the extra term \(\|\delta_F\|^2/(n-k_F)\), so the estimate tends to be too large. Cp’s calibration therefore relies on the fuller model being a plausible description of the mean structure, as well as on a common error variance.

Mean-Response Error and Future Responses

The derivation targeted error in estimating the vector of true mean responses. If the goal is instead to predict a new vector of responses at the same predictor values, write \(y^*=\mu+\varepsilon^*\), where the new errors have mean zero, variance \(\sigma^2\), and are independent of the errors used to fit the model. Then

The extra term is the irreducible error in the future observations. For models evaluated on the same \(n\) design points with the same error variance, it is common to all candidates. Thus Cp’s model comparison targets can be described as estimation error for the mean responses or, after adding that common term, expected error for future responses. The distinction matters for interpretation, but not for ranking those candidates by the criterion.

Common Mistakes and Practical Meaning

• Using residual sum of squares alone. A smaller model’s larger residual sum of squares does not by itself mean it predicts worse. Cp adds \(2k\hat{\sigma}^2_F\) to account for estimation variance and subtracts the common \(n\hat{\sigma}^2_F\) term.
• Counting only predictors. The \(k\) in the formula counts fitted parameters, including an intercept when present. The criterion’s penalty depends on the model’s dimension.
• Treating estimated Cp as an exact unbiased result. The exact expectation identity uses the true \(\sigma^2\). Replacing it with a fuller-model estimate is a practical plug-in step, with uncertainty of its own.
• Assuming any fuller model gives a valid variance estimate. If the fuller model has substantial lack of fit, its residual mean square can be inflated and the resulting Cp calibration can be misleading.

The derivation gives Cp a clear purpose: it estimates scaled prediction error by correcting residual fit for the fact that fitted values have been adjusted to the observed data. It does not guarantee that the best-ranked candidate is true, nor does it remove the need to check whether the regression assumptions and fuller-model variance estimate are reasonable.