Diagnosing Variation in Selected Penalty Values

Why Diagnose Variation in the Selected Penalty?

Cross-validation selects a penalty value by comparing prediction errors across candidate values. In high-dimensional genomic data, many predictors may be correlated and the prediction-error curve may be fairly flat near its minimum. As a result, changing which observations are assigned to folds can change the selected penalty even when predictive performance barely changes. Conversely, similar selected penalties do not guarantee that prediction error is stable.

Repeated cross-validation makes this variation visible. Its summaries need careful interpretation: repeating fold assignments on the same data measures sensitivity to those assignments, not the full sampling uncertainty that would arise from collecting a new study. This distinction matters when reporting a penalty choice or deciding whether a small change in prediction error is meaningful.

What Repeated Cross-Validation Summarizes

Suppose there are \(n\) independent subjects and a fixed set of candidate penalty values \(\lambda\). In one \(K\)-fold cross-validation repetition, the subjects are divided into \(K\) disjoint folds. For each candidate \(\lambda\), the model is fitted on \(K-1\) folds and its prediction loss is calculated on the held-out fold. Averaging held-out losses gives a cross-validation error for that candidate. The selected penalty for repetition \(r\) is the candidate with the smallest average error:

Here \(\Lambda\) is the candidate grid and \(\widehat{R}_r(\lambda)\) is the cross-validation estimate of prediction error in repetition \(r\). If several candidates tie, the tie-breaking rule must be specified in advance; for example, choosing the larger penalty is a common way to prefer greater regularization. The rule should be applied consistently across repetitions.

For a positive penalty, summaries should reflect its multiplicative scale. The arithmetic mean can be pulled upward by a few large values. The geometric mean is calculated by averaging log penalties and transforming back:

It is also useful to report the median, range, and, where meaningful, selected-penalty frequencies on the candidate grid. These summaries answer different questions: the median describes a typical run, the range reveals extremes, and frequencies show how often grid values are selected. If the candidate grid or preprocessing changes between runs, raw penalty values may not be comparable. Keep the grid, objective scaling, and preprocessing convention fixed when the goal is to diagnose fold-assignment variation.

Assumptions and a Useful Decomposition

Interpretation depends on how the validation procedure matches the prediction task. Ordinary random folds are appropriate when subjects are independent and the goal is prediction for new subjects drawn from the same population. If observations are clustered by family, site, or repeated measurements, folds should keep related observations together when the target is prediction for new independent clusters. If prediction is for a later time period, random folds can leak information across time; a time-ordered split is more appropriate.

All data-dependent preprocessing must be estimated using only the training portion of each fold. This includes operations such as centering and scaling predictors, filtering features, and any outcome-informed screening. Applying such operations once to the full data before cross-validation allows held-out observations to influence the fit and can make the reported errors too optimistic. The penalty grid and the scale of the fitting objective must also be consistent across repetitions.

The distinction between fold and sampling variation can be expressed with the law of total variance. Let \(D\) denote a random study sample and \(F\) a random fold assignment conditional on that sample. For a quantity \(T(D,F)\), such as a cross-validation error at a fixed penalty, the total variance is:

The first term describes variation across samples in the average result over fold assignments. The second describes variation across fold assignments within a sample, averaged over samples. Repeating folds on one observed data set provides information about the conditional fold variation, not a direct estimate of both terms. For the selected penalty, which is a nonlinear function of the errors, the same conceptual distinction applies, although this variance identity does not make the distribution of selected penalties normal or guarantee a simple standard error.

Worked Summaries

Diagnostics, Alternatives, and Limitations

A useful report pairs the distribution of selected penalties with the distribution of prediction errors. A wide penalty range with nearly unchanged errors often indicates that the error curve is shallow around its minimum: the data distinguish predictive performance poorly among nearby candidates. A narrow penalty range with widely changing errors calls for checking fold construction, preprocessing, data leakage, influential observations, and numerical convergence. These patterns are clues, not proofs of a particular cause.

Other checks include confirming that every run used the intended folds and the same candidate grid, plotting error against \(\log(\lambda)\) for each repetition, and checking whether minima occur at the grid boundaries. Repeated boundary selections may mean the candidate grid is too narrow. If a candidate is selected because of a tie or tiny numerical differences, report the tie rule and inspect the neighboring errors rather than implying meaningful precision in \(\lambda\).

Repeated cross-validation has practical advantages over a single split: it reveals sensitivity to fold assignment without discarding most of the data for one permanent validation set. Its limitations are computational cost and dependence between runs, since the same subjects are reused. The run-to-run standard deviation is descriptive; dividing it by the square root of the number of repetitions does not generally produce a valid standard error for population prediction performance.

When sampling variation is the concern, resampling subjects or collecting an independent sample changes the data as well as the folds. Subsampling or bootstrap designs can be informative, but require careful attention to dependence and to the prediction target. For estimating predictive performance after tuning, nested cross-validation uses inner folds to choose the penalty and outer held-out folds to assess the entire tuning procedure. An external test set is another option when available. Neither repeated inner cross-validation errors nor their minimum alone should be treated as an unbiased final performance estimate: choosing the minimum among noisy candidate errors tends to make it optimistic.

The one-standard-error rule, introduced in the earlier material on cross-validation, changes the selection rule by choosing a simpler fit within a stated error tolerance. Repeating that rule can also produce a distribution of selected penalties, but the rule does not remove fold or sampling variation. The purpose here is to diagnose and report that variation, not to turn a spread of penalty values into a claim about which genes are reliably selected.

Common Mistake