Cross-Validation for GLM Penalty Selection

Why Cross-Validation Is Needed

A penalized GLM fit depends on the penalty strength: a large penalty generally shrinks coefficients more, while a small penalty allows a closer fit to the training data. The training likelihood or deviance alone cannot determine which penalty will predict well. As the penalty weakens, training fit can improve even when prediction on new observations does not. K-fold cross-validation estimates predictive loss on observations excluded from each fit, allowing penalty values to be compared on a common basis.

The essential discipline is to treat each fold as a small prediction study. For a given fold, estimate every data-dependent part of the procedure using only the training observations, then apply that fitted procedure to the held-out observations. This includes predictor standardization and any other preprocessing, as well as the penalized GLM coefficients.

Definition and Calculation

Definition: Partition the $n$ observations into $K$ nonempty folds $I_1,\ldots,I_K$, with sizes $n_k=|I_k|$ and $\sum_{k=1}^K n_k=n$. For fold $k$, fit the penalized GLM using observations outside $I_k$. At penalty value $\lambda$, let $\hat\mu_i^{(-k)}(\lambda)$ be the fitted mean for held-out observation $i\in I_k$, obtained using that fold's fitted preprocessing and coefficients. Given a per-observation loss $L(y,\mu)$, the $K$-fold cross-validation score is $$ \operatorname{CV}(\lambda)=\frac{1}{n}\sum_{k=1}^K\sum_{i\in I_k}L\!\left(y_i,\hat\mu_i^{(-k)}(\lambda)\right). $$ The selected penalty is a value in the candidate set $\Lambda$ minimizing this score.

For unequal fold sizes, this definition weights each observation equally. Equivalently, if $\bar L_k(\lambda)$ is the mean held-out loss in fold $k$, then the score is the fold-size-weighted mean of those fold means. The loss must be specified: common choices include negative log-likelihood contributions or unit deviance contributions. Lower values indicate better held-out performance under the chosen scoring rule.

For example, the Bernoulli negative log-likelihood loss is $L(y,\hat p)=-y\log(\hat p)-(1-y)\log(1-\hat p)$, where $y\in\{0,1\}$. For a Poisson response, the unit deviance contribution is $2\{y\log(y/\hat\mu)-(y-\hat\mu)\}$ for $y>0$, and $2\hat\mu$ for $y=0$. These scores assess predictions on their respective scales; the chosen loss should match the predictive target and be applied consistently across folds and penalty values.

Worked Example: Aggregating Unequal Fold Sizes

Suppose five held-out observations are divided into two folds. The first fold has three observations with losses $0.2, 0.5, 0.8$; the second has two observations with losses $0.3, 0.7$. Compute the cross-validation score.

The total loss in the first fold is $0.2+0.5+0.8=1.5$, and its mean is $1.5/3=0.5$. The second fold's total is $0.3+0.7=1.0$, and its mean is $1.0/2=0.5$. The observation-weighted score is $$ \operatorname{CV}=\frac{1.5+1.0}{5}=0.5. $$ Here both fold means happen to agree, so the unweighted average of fold means also equals $0.5$. In general, it does not: the correct fold-mean expression is $(3\bar L_1+2\bar L_2)/5$, not $(\bar L_1+\bar L_2)/2$.

Worked Example: Comparing Two Logistic Penalties

Across four equal-sized held-out folds, suppose one candidate penalty produces observed Bernoulli outcomes and predicted probabilities as follows: $(y,\hat p)=(1,0.65),(0,0.60),(1,0.55),(0,0.60)$. Compute its mean held-out log loss.

For an outcome of 1, the loss is $-\log(\hat p)$; for an outcome of 0, it is $-\log(1-\hat p)$. Thus the four losses are $-\log(0.65)\approx0.4308$, $-\log(0.40)\approx0.9163$, $-\log(0.55)\approx0.5978$, and $-\log(0.40)\approx0.9163$. Their unrounded total is approximately $2.8612$, so the mean is approximately $2.8612/4=0.7153$.

For a second penalty, suppose the four losses are $0.60,0.75,0.62,0.70$. Their mean is $(0.60+0.75+0.62+0.70)/4=2.67/4=0.6675$. This second penalty has the lower cross-validation log loss, $0.6675$ versus $0.7153$, and is preferred by this criterion. The comparison is about held-out prediction, not which fit has the larger training likelihood.

What the Cross-Validation Average Estimates

The following result explains the target of ordinary K-fold cross-validation. Assume the observations are independent and identically distributed, the folds are fixed in advance or assigned independently of the data, and the fitting procedure—including preprocessing—is applied in the same way to every training set. Fix $\lambda$ and suppose the loss has finite expectation. Write $m=n-n_k$ for a training-set size in a fold. The result is stated for equal fold sizes, so $m$ is the same for every fold.

Result: For a fixed penalty value, the expected K-fold score equals the expected loss on an independent observation from the same distribution when the fitting procedure is trained on $m=n(1-1/K)$ independent observations: $$ E[\operatorname{CV}(\lambda)] = E\!\left[L\!\left(Y_{\mathrm{new}},\hat\mu_{\mathcal T}(X_{\mathrm{new}};\lambda)\right)\right]. $$ Here $\mathcal T$ denotes an independent training sample of size $m$, and $(X_{\mathrm{new}},Y_{\mathrm{new}})$ is an independent observation.

To verify the claim, fix a fold $k$. Its training observations and its held-out observations are disjoint subsets of an independent sample, so the held-out observations are independent of the training data. By identical distribution, each held-out pair has the same distribution as a fresh observation $(X_{\mathrm{new}},Y_{\mathrm{new}})$, independent of that fold's training sample. Each fold-specific prediction therefore has the same expected loss as the displayed fresh-observation loss. Averaging the $n$ held-out losses preserves that expectation by linearity:

E[\operatorname{CV}(\lambda)] =\frac{1}{n}\sum_{k=1}^K\sum_{i\in I_k} E\!\left[L\!\left(y_i,\hat\mu_i^{(-k)}(\lambda)\right)\right] =E\!\left[L\!\left(Y_{\mathrm{new}},\hat\mu_{\mathcal T}(X_{\mathrm{new}};\lambda)\right)\right].

This is an expectation statement for a fixed $\lambda$, not a guarantee that the realized score equals the prediction risk. It also targets a procedure trained on fewer than $n$ observations. Finally, minimizing many noisy scores tends to favor a candidate whose score happened to be unusually low; the minimum score is not automatically an unbiased assessment of the final selected procedure.

Preprocessing Must Be Learned Within Each Fold

Standardizing predictors before penalization is often useful, but the means and scales are estimated from data. In cross-validation they are part of the fitted procedure, not fixed facts about the population. For each fold, calculate the transformation from the training observations, apply those same values to both training and held-out predictors, fit the penalized GLM on the transformed training data, and predict on the transformed held-out data. The held-out outcomes must not enter coefficient fitting or any preprocessing decision.

Worked Example: Standardization Without Leakage

Consider one predictor with training-fold values $0$ and $2$, and held-out value $10$. Use the training mean and the sample standard deviation, with denominator $r-1$ for a training size $r=2$.

The training mean is $\bar x_{\mathrm{train}}=(0+2)/2=1$. The training sample variance is $\{(0-1)^2+(2-1)^2\}/(2-1)=2$, so the training standard deviation is $\sqrt 2$. The transformed training values are $(0-1)/\sqrt2=-1/\sqrt2$ and $(2-1)/\sqrt2=1/\sqrt2$. The held-out value must use the same training-fold quantities: $(10-1)/\sqrt2=9/\sqrt2\approx6.364$.

If the held-out predictor were incorrectly included when estimating the mean and sample standard deviation, the mean would be $4$, and the sample standard deviation would be $\sqrt{\{(0-4)^2+(2-4)^2+(10-4)^2\}/2}=\sqrt{56/2}=\sqrt{28}$. The held-out standardized value would then be $6/\sqrt{28}\approx1.134$. This transformation has used information from the held-out fold and is not the procedure that would be available when predicting a genuinely new observation. The numerical difference also shows why leakage can materially alter penalized fitting: a penalty acts on coefficient scales, so changing predictor scaling changes the effective fit.

Penalty Paths and a Practical Procedure

A candidate set $\Lambda$ should have a consistent meaning across folds. A common approach is to specify a shared grid of absolute $\lambda$ values in advance and fit every value on every fold's training data. If instead a path is parameterized relative to a fold-specific $\lambda_{\max}$, the same path index or fraction need not represent the same absolute penalty across folds. Such a design can be used, but the quantity being compared must be stated clearly; fold-specific path positions should not be mistaken for identical penalty values.

Likewise, do not use all observations to choose a data-dependent grid, transformation, or screening decision if that choice uses information that should be held out. For a fixed shared grid, fit each candidate using only the training fold. Warm starts can reduce computation along a path, but they must not carry fitted information from a different fold's held-out observations into the current fit.

Set the prediction task.
Choose the response family, offset or exposure treatment when applicable, and a held-out loss appropriate to the target.

Specify folds and candidate penalties.
Assign observations to folds and choose a comparable candidate set. Keep any grouping or dependence structure in the data in mind when assigning folds.

Fit within each training fold.
Estimate preprocessing from training predictors only, transform training data, and fit the penalized GLM for each candidate penalty.

Predict and score held-out observations.
Apply the training-fold transformation and fitted model to the held-out predictors; compute one loss per held-out observation.

Aggregate and select.
Average all held-out losses with equal weight per observation, then select a penalty minimizing the resulting score.

Common Mistakes and Why This Matters

Averaging fold means equally when fold sizes differ. This gives a small fold the same influence as a large one. Aggregate observation-level losses, or weight each fold mean by its size.
Standardizing once on the complete data. This lets held-out predictors influence the transformation. Re-estimate means, scales, and other data-dependent preprocessing within each training fold.
Using training loss to select the penalty. Training loss generally rewards a closer fit and does not provide the held-out comparison cross-validation is designed to supply.
Comparing different penalty meanings as if they were identical. A fold-specific path index can correspond to different absolute penalty values. State the grid and its interpretation.
Calling the minimum CV score an unbiased final performance estimate. The same scores were used to select the penalty, so selection can make the minimum optimistic. An untouched test set or a separate resampling assessment can be used when an honest final performance estimate is needed.

Cross-validation is useful because it compares penalized GLMs on predictions from observations excluded from their respective fits. Its validity depends on preserving that separation throughout the analysis: not only for coefficient estimation, but also for preprocessing and other data-driven steps. The score is a practical estimate of predictive loss for a specified fitting procedure and training size; it is not a universal property of a penalty value detached from the model-building process.

Check Your Understanding

Answer each question using the definitions and procedures above.

Why is the observation-weighted score preferable to the unweighted average of fold means when fold sizes differ?
For a Bernoulli GLM, write the per-observation negative log-likelihood loss for $y=0$ and for $y=1$.
A predictor is centered and scaled using all observations before cross-validation. What information leaks, and how should the procedure be changed?
What training-sample size does equal-sized $K$-fold cross-validation target for each fitted model?
Why might the minimum cross-validation score be optimistic as an estimate of the selected procedure's final predictive performance?