What Is Held Fixed?
An external cohort can be used to ask whether a discovery result carries over beyond the data that produced it. But “apply the discovery result” can mean different things. A genuinely frozen analysis carries the prespecified feature set, transformations, model form, fitted parameters, and decision rule into the external cohort unchanged. A refit analysis estimates at least some of those choices or parameters again using external-cohort data. The distinction is not cosmetic: it determines what the analysis can establish.
Suppose the discovery data are denoted by \(D\), and the external validation data by \(V\). A discovery analysis uses \(D\) to produce a complete rule \(f_D\), which maps available measurements to a score, prediction, or classification. In a frozen analysis, \(f_D\) is applied to measurements in \(V\) without using validation outcomes to alter the rule. In a refit analysis, some part of \(f_D\) is replaced by a rule estimated or selected using \(V\).
“Frozen” does not mean that no computation is done on the external measurements. The rule must be applied to those measurements, and technical steps needed to put measurements on the defined scale may be required. What matters is whether those steps were specified in advance and whether they alter the learned rule. For example, applying discovery-set means and standard deviations to standardize validation measurements is compatible with a frozen rule. Recalculating those means and standard deviations from the validation cohort changes the transformation and therefore changes what is being assessed.
Assumptions for a Frozen Validation
A frozen result supports an external-validation claim only when the rule and its application are sufficiently well defined. The following conditions should be checked before interpreting the result.
- Independent data use. The external cohort must not have contributed participants or outcome information to the discovery choices being evaluated. Otherwise, the evaluation is not independent of the development process.
- Operationally defined rule. The feature list, measurement units, preprocessing, missing-data handling, fitted parameters, and any threshold must be specified well enough that another analyst could apply the same rule.
- Comparable measurements. The external variables must correspond to the intended discovery features on a usable scale. If measurement platforms differ, a prespecified harmonization procedure may be necessary; the earlier material on cross-platform measurement comparability addresses this issue.
- Appropriate target cohort. The validation cohort must represent the population to which the claim refers. A successful application in a selected cohort does not by itself establish performance in populations with different eligibility criteria or measurement conditions.
- No outcome-guided revision during evaluation. If validation outcomes are used to select features, tune parameters, choose a threshold, or decide which result to report, the resulting analysis is no longer a frozen test of the discovery rule.
These conditions do not guarantee that a discovery rule will work externally. They make the test interpretable: the external data are being used to evaluate a rule that was already defined, rather than to construct a better-fitting replacement.
Worked Example: Applying a Frozen Multi-Omics Score
Suppose a hypothetical discovery analysis defined a two-feature score. Feature \(A\) is a transcript measurement and feature \(B\) is a protein measurement. The discovery means and standard deviations were 5 and 2 units for \(A\), and 10 and 4 units for \(B\). The locked rule is
For one hypothetical external participant, the measured values are \(A=8\) units and \(B=14\) units. Applying the discovery transformations gives
The score is therefore
Because \(1.3\geq 1\), this participant is classified as high by the frozen rule. The calculation uses the discovery means, standard deviations, weights, and threshold. It does not estimate a new score from external outcomes. To apply this rule validly, the external measurements must use compatible units or a prespecified conversion, and the required measurements must be available; otherwise, the score as defined cannot be applied without an explicit change to the protocol.
What Refitting Changes
Refitting can occur at several stages. It includes selecting different molecular features, recalculating normalization parameters, estimating new regression coefficients, tuning a penalty or model complexity, or choosing a new classification threshold using external outcomes. Even if the same software and broad model family are used, the resulting analysis is refit when any relevant component is learned again from the validation cohort.
Some changes are necessary for technical reasons, but they should not be hidden under the word “frozen.” If external-cohort data are used to estimate a transformation, the analysis evaluates a pipeline that includes that cohort-adaptive step, not the exact discovery pipeline. If outcomes are also used, the outcome-guided component is particularly important to disclose because it can make the analysis fit the validation cohort.
A refit may answer a worthwhile question: can a model with this feature set be estimated in the external cohort, or do the available data support a locally adapted model? But that is different from asking whether the discovery-defined rule transports unchanged. If feature selection and fitting are both repeated, the result concerns a new model-building exercise in the external cohort. A good fit to that cohort is not, by itself, evidence that the discovery model validated.
Worked Example: Re-estimating a Model in the External Cohort
A hypothetical discovery analysis proposed a linear model relating a molecular score \(x\) to an outcome \(y\), with the locked discovery equation \(\hat{y}=1+2x\). In a separate external cohort, suppose the observed values are \((x,y)=(0,1),(1,2),(2,2),(3,5)\). An analyst now fits a new intercept and slope by ordinary least squares. This is a refit, because the model parameters are estimated from the external outcomes.
The external means are \(\bar{x}=(0+1+2+3)/4=1.5\) and \(\bar{y}=(1+2+2+5)/4=2.5\). The least-squares slope is the sum of cross-products about the means divided by the sum of squared deviations in \(x\):
The intercept is \(\hat{\beta}_0=\bar{y}-\hat{\beta}_1\bar{x}=2.5-(1.2)(1.5)=0.7\). Thus the refitted equation is \(\hat{y}=0.7+1.2x\), not the frozen discovery equation \(\hat{y}=1+2x\). For \(x=2\), the frozen equation predicts \(1+2(2)=5\), whereas the refitted equation predicts \(0.7+1.2(2)=3.1\). The difference is not a computational inconsistency: the equations use different parameters. The refit describes a relationship estimated from this hypothetical external sample; it does not demonstrate that the discovery equation carried over unchanged.
This calculation illustrates the distinction, not a complete inferential analysis. To interpret the fitted line as a linear conditional-mean model, the model form and relevant error assumptions would need justification; no such assumptions or population conclusion are established by these four observations alone.
Why a Frozen Evaluation Is a Different Claim
The basic justification for freezing is that, conditional on the discovery data, the rule \(f_D\) is fixed. If the validation observations are independent of \(D\) and drawn from the target population, an average validation loss evaluates that fixed rule in that population, subject to sampling variability. For a loss function \(L\), its target is the expected loss of the discovery rule, not the loss of a rule newly optimized on the validation outcomes.
Here \(X_i\) denotes the measurements used by the rule, \(Y_i\) the relevant outcome, and \(n_V\) the number of eligible validation participants. If the validation sample is representative and independent of discovery, the sample average is an estimate of the fixed rule’s expected loss in the target population. If the analyst instead chooses \(f\) to minimize the same validation loss, the reported loss reflects both the rule and the fitting process. It no longer estimates performance of the unchanged discovery rule.
This distinction is about the claim, not about whether one analysis is universally superior. A frozen analysis is direct evidence about the prespecified discovery procedure. A refit may be appropriate for model adaptation or exploratory development, but needs a separate, honest evaluation if its performance is to be assessed. For example, after refitting on one external cohort, a further independent cohort can assess the newly estimated rule without reusing the data that produced it.
Choosing and Reporting the Analysis
The analysis should follow the claim of interest. For direct external validation, lock the rule before examining validation outcomes and apply it as specified. For local adaptation, state clearly which components were re-estimated and describe the work as refitting or model development, not as unchanged validation. A useful report identifies the feature set, transformations, parameter values, missing-data rules, threshold, and every component updated in the external cohort.
| Analysis choice | What is held fixed? | Claim it can support | Main caution |
|---|---|---|---|
| Frozen application | Feature set, transformations, parameters, and decision rule | Whether the specified discovery rule applies in the external cohort | Requires compatible measurements and an untouched rule |
| Partial refit | Some components, such as the feature list, remain fixed; others are re-estimated | Whether the specified components can be adapted in the external cohort | State exactly which components changed; it is not an unchanged-rule test |
| Full refit | Only the broad model-building goal may remain fixed | Whether a new model can be developed from the external data | Assessing fit on the same data used to build the model is optimistic |
A practical diagnostic is to ask, for every analysis step, “Could this step produce a different rule if the validation outcomes changed?” If the answer is yes, then the step is outcome-adaptive and must be locked in advance for a frozen outcome-based evaluation. Also check that the feature identifiers map to the intended molecules, units and assay scales match the protocol, no outcome-informed exclusions were introduced, and no participants overlap discovery and validation. Unavailable features, excessive missingness, or incompatible scales are not reasons to silently substitute a new feature set; they are limitations to report or grounds for a prespecified alternative analysis.
Common Pitfall and Limitations
Wrong reasoning: “We used the discovery feature list, so this is a frozen validation,” even though the external outcomes were used to recalculate coefficients and select a threshold. Correction: a fixed feature list alone does not freeze the whole analysis. Re-estimating coefficients or choosing a threshold from validation outcomes makes the analysis a refit, and the claim must reflect that.
The frozen/refit distinction does not eliminate problems of cohort comparability, measurement error, missing data, or limited sample size. A frozen analysis may fail because the target population or platform differs, and a refit may be unstable when the external cohort is small relative to the number of candidate features. Neither type of analysis, by itself, establishes causation. In either case, transparently reporting what was fixed, what was estimated, and what population was studied is essential to interpreting the evidence.
Worked Example: Feature Selection Changes the Validation Question
Suppose a hypothetical discovery analysis selected features \(A\) and \(B\) and locked the score \(S=0.8A+0.2B\), with all transformations and a decision threshold specified in advance. In the external cohort, an analyst examines the outcome and finds that feature \(C\), which was not selected in discovery, is more strongly associated with the outcome. The analyst replaces \(A\) and \(B\) with \(C\) and reports the performance of a newly fitted \(C\)-based rule.
This is not a frozen application: the selected feature set changed from \(\{A,B\}\) to \(\{C\}\), and the change used validation outcomes. The reported result may be useful as exploratory model development, but it does not test the locked discovery score. A valid frozen analysis would apply \(0.8A+0.2B\), using the prespecified transformations and threshold, if those measurements are available and comparable. If they are not, the limitation should be reported rather than silently replacing the features.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A discovery score uses three features, fixed weights, and discovery-set standardization parameters. The analyst uses those parameters but chooses a new threshold after inspecting external outcomes. Is the analysis frozen?
- Why does estimating a new set of coefficients in the external cohort change the claim, even if the feature list is unchanged?
- A frozen score cannot be calculated for some participants because one required assay is unavailable. May the analyst replace that feature with a correlated one and still call the analysis frozen?
- What is the key independence condition that allows the validation average loss to estimate the expected loss of a fixed discovery rule?
Solutions. (1) No. The features, weights, and standardization are fixed, but choosing the threshold from external outcomes changes the decision rule and makes the analysis partly refit. (2) The coefficients are part of the rule mapping measurements to predictions. Re-estimating them means the external outcomes helped construct the evaluated rule, so the analysis no longer tests the discovery equation unchanged. (3) No. Substitution changes the feature definition. The analyst should report the missing assay and use a replacement only under a clearly specified alternative analysis; that alternative is not the same frozen score. (4) The validation participants must be independent of the discovery data used to define the rule and must represent the target population for the claim. Under that condition, the fixed rule’s validation losses estimate its expected loss in that population.