Two Different Questions Require Two Different Summaries
A penalized genomic model can predict well without selecting the same genes every time its training sample changes. Conversely, a small set of genes may be selected consistently while predictions remain only moderately accurate. A useful report therefore treats predictive performance and selection stability as separate results, and states how each was evaluated.
The report should make it possible to understand what data were used for tuning, what observations were held out for evaluation, which penalty was chosen, and what counts as a selected feature. Its numbers should be accompanied by enough design detail to interpret them. A test-set score alone does not describe how stable the selected genes are; a list of frequent genes alone does not show whether the model predicts accurately.
Set Up the Evaluation Before Reporting Numbers
For an honest estimate of predictive performance, the evaluation data must not have been used to fit the model, choose the penalty, choose features, or make preprocessing decisions. The training data should contain all steps that learn from data, including centering, scaling, imputation, and penalty tuning. Apply those learned transformations to the test data without refitting them there. This separation prevents information from the test outcomes from influencing the predictions being evaluated.
The test observations should also represent the prediction setting of interest. For example, if the intended use is prediction in new patients, randomly dividing repeated measurements from the same patient between training and test sets can make performance look too favorable. A split by patient, family, or collection site may be needed when those are the independent units relevant to deployment. The interpretation of a test score relies on the test data being suitably independent of the fitting process and representative of the intended target population.
Penalty choice belongs inside the training process. In cross-validation, fit the candidate penalized models on each training fold, predict the corresponding held-out fold, and aggregate the fold errors according to the prespecified rule. If feature selection stability is evaluated using perturbations, define the perturbation scheme, fitting procedure, penalty rule, and selection rule. If the penalty is reselected within every perturbation, say so; if it is held fixed, report how it was fixed. Those choices can change the resulting frequencies.
Choose metrics that answer the scientific prediction question. For continuous outcomes, RMSE and MAE summarize prediction errors in the outcome's units. \(R^2\) compares squared prediction error with variation around the test-set mean; it can be negative when predictions are worse than using that mean as a benchmark. No single metric is universally best, and the test-set size and uncertainty should be reported where possible.
Worked Examples
Worked Example: Computing Test-Set Prediction Metrics
Suppose a hypothetical test set contains four patients. Their measured expression outcome is in arbitrary assay units, and the model's predictions are in the same units.
| Patient | Observed | Predicted | Prediction error |
|---|---|---|---|
| A | 2 | 3 | -1 |
| B | 4 | 3 | 1 |
| C | 5 | 6 | -1 |
| D | 7 | 6 | 1 |
The errors are observed minus predicted. The mean squared error is \((1+1+1+1)/4=1\), so the root mean squared error is \(\sqrt{1}=1.00\) assay unit. The mean absolute error is \((1+1+1+1)/4=1.00\) assay unit.
The observed mean is \((2+4+5+7)/4=4.5\) assay units. The total sum of squares about that mean is \( (2-4.5)^2+(4-4.5)^2+(5-4.5)^2+(7-4.5)^2=13\). The residual sum of squares is \(4\), so test-set \(R^2=1-4/13=0.6923\). For these four observations, the model's squared prediction error is about 69.2% lower than the squared error from predicting the test-set observed mean. This small hypothetical test set provides an illustrative point estimate, not a precise guarantee of performance on future patients.
Worked Example: Reporting a Penalty Choice
In a hypothetical training set, five-fold cross-validation compares two penalty values. The reported error is the mean fold RMSE in outcome units. Suppose the estimated minimum is 0.92 at \(\lambda=0.08\), with estimated standard error 0.12. At \(\lambda=0.12\), the mean RMSE is 0.96 and its estimated standard error is 0.09. Assume the analysis prespecified the one-standard-error rule and that the larger penalty gives a sparser fit.
The one-standard-error cutoff from the minimum is \(0.92+0.12=1.04\) outcome units. Since \(0.96\leq1.04\), the fit at \(\lambda=0.12\) is within one estimated standard error of the minimum. Under the stated rule, choose the larger penalty, \(\lambda=0.12\), for its sparser fit. A clear report gives both the selected penalty and the rule: “Five-fold cross-validation selected \(\lambda=0.12\) using the one-standard-error rule; its mean RMSE was 0.96 outcome units, compared with the minimum 0.92 at \(\lambda=0.08\).”
These cross-validation errors guide model choice; they are not an independent final test-set estimate. If the same folds are used both to choose the penalty and to report performance, the reported minimum can be optimistic because it was selected from competing estimates. A separate held-out test set, or a design that accounts for tuning during evaluation, is needed for an external performance estimate.
Worked Example: Reporting Selection Frequencies and Their Precision
Suppose 1,000 independently generated subsamples are drawn from a hypothetical genomic training data set, and the entire stated selection procedure is applied to each. A gene is counted as selected when its fitted coefficient is nonzero. Gene G1 is selected in 800 subsamples, G2 in 430, and G3 in 90.
The frequencies are \(800/1000=0.800\), \(430/1000=0.430\), and \(90/1000=0.090\). Thus G1 was selected in 80.0% of these perturbations, G2 in 43.0%, and G3 in 9.0%. For G1, the conditional Monte Carlo standard error of the estimated frequency is approximately \(\sqrt{0.800(1-0.800)/1000}=\sqrt{0.00016}=0.0126\), or 1.26 percentage points, if perturbation draws are independent under the specified resampling scheme.
That Monte Carlo standard error describes simulation precision for estimating the frequency under this procedure and this observed data set. It does not quantify all uncertainty about which genes would be selected in a new study. The report should name the subsampling fraction, whether the penalty was reselected in each subsample, and the nonzero-coefficient rule. For example: “Under 1,000 independent 80% subsamples, with the penalty reselected by the prespecified cross-validation rule in each subsample, G1 was selected 800 times (80.0%).”
What Selection Frequencies Do and Do Not Mean
A frequency is conditional on the analysis choices that generated it. Changing the subsample size, penalty-selection rule, preprocessing, or definition of selection can change \(\hat{\pi}_j\). Report those choices alongside the frequencies, not only in a distant methods description. Giving counts as well as percentages makes the denominator transparent.
Selection is also complicated by correlation among genomic features. If two genes carry similar predictive information, a penalized procedure may select one in some perturbations and the other in others. Each gene can then have a modest frequency even when the signal represented by the pair is repeatedly captured. Conversely, a high frequency does not establish a causal role, a reliable effect size, or replication in an independent cohort. Interpret individual-gene frequencies in light of the fitted procedure and the dependence structure of the features.
Do not treat a threshold such as 80% as universally meaningful. If a threshold is used to define a reported stable set, state the threshold and how it was chosen. A frequency table may be more informative than a binary “stable/unstable” label, especially near the chosen cutoff. Include the number of features examined or summarize how many met prespecified frequency ranges so readers can assess the scale of selection.
Constructing a Concise Results Statement
A compact report can still be reproducible. Include the outcome and prediction target, the test or cross-validation design, the metric and units, the penalty and selection rule, and the feature perturbation details. Identify whether any performance estimate is from data independent of tuning. When space allows, provide a table with one row per model or metric, and a separate table or list of feature frequencies.
For example, a hypothetical report might read: “The elastic-net penalty was chosen by five-fold cross-validation using the one-standard-error rule. On an independent test set of 60 patients, RMSE was 1.14 expression units and MAE was 0.87 expression units. Across 1,000 prespecified 80% subsamples, with the penalty reselected within each subsample, G1 was selected in 80.0% of fits and G2 in 43.0%. These frequencies describe stability under the stated resampling procedure, not evidence of causal effects.” The numerical values here are hypothetical; the sentence illustrates how to keep the claims within what the analysis supports.
When there is no independent test set, say so. Cross-validation can be used to estimate predictive performance, but the report must explain whether the same cross-validation results also selected the penalty. Do not label a tuning score as an independent test result. A single train/test split is simpler, while repeated cross-validation can use data more efficiently; both depend on an appropriate splitting design, and neither removes uncertainty from a limited or unrepresentative sample.
Diagnostics, Alternatives, and Common Pitfalls
Check that every test observation was excluded from all data-adaptive steps, that fold assignments respected the independent sampling units, and that the stated performance metric matches the outcome scale and intended use. For repeated resampling, check that the number of perturbations is large enough for the reported frequencies to have adequate Monte Carlo precision. Inspect whether the features with intermediate frequencies exchange selection with correlated features; reporting only the most frequent genes can conceal that pattern.
A major pitfall is to write “G1 has an 80% probability of being important” because it was selected in 80% of subsamples. That reasoning is wrong: 80% is the observed resampling frequency under a specific procedure, not a posterior probability of biological importance. The correction is to report exactly that frequency and its conditioning scheme, and to avoid biological or causal claims that the resampling result does not establish.
Another mistake is to report the lowest cross-validation error after trying many penalties as though it were an unbiased test result. The correction is to distinguish the tuning criterion from evaluation on independent data, or use an evaluation design that accounts for tuning. Likewise, reporting “the model achieved \(R^2=0.69\)” without identifying the test data, outcome units, or evaluation design leaves readers unable to judge the estimate.
The main alternatives serve different purposes. A single held-out test set gives a direct, interpretable evaluation but may be noisy when the sample is small. Cross-validation uses training data more efficiently for tuning, but its minimum error is not automatically an independent performance estimate. Repeated subsampling or bootstrap perturbations characterize how selection changes under the chosen resampling scheme; they do not replace external replication. In high-dimensional genomic settings, limited sample size, correlated features, and distribution shifts can make both prediction estimates and feature frequencies unstable.
Specify the outcome, units, and intended population or prediction setting.
State which data informed preprocessing and penalty choice, and identify data held out from those steps.
Give the metric, value, units, evaluation design, and test-set size when applicable.
Give the perturbation scheme, selection definition, denominator, and feature frequencies; label their interpretation as procedure-specific.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A model has test-set RMSE of 1.3 expression units. What does this describe, and what does it not establish?
- A gene is selected in 60 of 200 subsamples. Calculate its selection frequency and state an appropriate interpretation.
- Why should test outcomes not be used to choose the penalty or preprocessing transformations?
- Cross-validation gives a minimum RMSE of 0.80, while the independent test RMSE is 0.95. Which value should be identified as the test-set performance?
- Two correlated genes have selection frequencies of 45% and 40%. Give one reason not to conclude that their shared predictive signal is absent.
Solutions. (1) The model's root mean squared prediction error on the evaluated test observations is 1.3 expression units. It does not guarantee the same error in a future population or establish that selected genes are biologically causal. (2) The frequency is \(60/200=0.300\), or 30.0%; the gene was selected in 30.0% of the specified subsamples, not “important with probability 30%.” (3) Using test outcomes in fitting or tuning lets information from evaluation data affect the model, compromising the independence of the test assessment and potentially making performance look too favorable. (4) The independent test-set RMSE, 0.95, is the test performance; 0.80 is the minimum cross-validation tuning score. (5) The procedure may select one correlated feature in place of the other across perturbations, so neither individual frequency alone shows whether their shared predictive information is absent.