What Does a Performance Score Describe?
A model can perform well in one cohort and less well in another even when its code and fitted parameters are unchanged. In multi-omics prediction, cohorts may differ in patient mix, disease prevalence, recruitment criteria, assay platforms, clinical practice, or the relationship between molecular measurements and outcomes. A score is therefore not an unconditional property of a model: it describes performance for a particular population, outcome definition, prediction time, and evaluation procedure.
Let \(X\) denote the predictor vector, such as a set of measured omics features and clinical covariates, and let \(Y\) denote the outcome. Write \(P_{\mathrm{tr}}\) for the training-population distribution and \(P_{\mathrm{dep}}\) for the deployment-population distribution. For a fixed fitted prediction rule \(f\) and a loss function \(L\), define its deployment risk as the expected loss for a randomly selected deployment individual:
For binary classification, \(L\) might be the indicator of a wrong class prediction; for probability predictions, it could be a probability score such as the Brier score. Different metrics answer different questions, so “performance” must be tied to the specified metric. The earlier material on choosing metrics and evaluating classifiers under imbalance is relevant here: this tutorial asks how population changes alter the interpretation of those metrics.
Three Ways the Populations Can Differ
A useful diagnosis separates changes in predictor distributions, outcome prevalence, and predictor–outcome relationships. These are distinct descriptions of a distributional change, not mutually exclusive explanations. More than one can occur at once.
| Change | What differs? | Why it matters |
|---|---|---|
| Predictor-distribution shift | \(P_{\mathrm{dep}}(X)\) differs from \(P_{\mathrm{tr}}(X)\) | The deployment population may contain different predictor profiles, which can change average performance even if conditional performance is stable. |
| Outcome-prevalence shift | \(P_{\mathrm{dep}}(Y)\) differs from \(P_{\mathrm{tr}}(Y)\) | Metrics that depend on the outcome mix, including accuracy and predictive values, may change. |
| Predictor–outcome relationship shift | \(P_{\mathrm{dep}}(Y\mid X)\) differs from \(P_{\mathrm{tr}}(Y\mid X)\) | The same predictor profile may carry different outcome risk, so performance conditional on \(X\) may change. |
The phrase covariate shift is commonly used for a particular case: \(P(X)\) changes while \(P(Y\mid X)\) remains invariant. Label shift commonly means that \(P(Y)\) changes while \(P(X\mid Y)\) remains invariant. A changed \(P(Y\mid X)\) is often called concept shift. These conditions are not interchangeable. For example, a difference in outcome prevalence alone does not prove label shift, because prevalence can also change when the predictor mix changes.
A model can have the same sensitivity and specificity in two populations yet have different positive predictive values because the outcome prevalence differs. Conversely, even similar predictor distributions do not guarantee similar performance if the outcome relationship has changed. Comparing only the marginal distributions of the measured predictors cannot settle whether the model transports.
When Can Performance Be Reweighted?
Under covariate shift, a training sample can sometimes be used to estimate deployment risk by weighting observations according to how common their predictor profiles are in deployment relative to training. This approach requires more than observing different predictor histograms. The outcome relationship must be stable, and deployment predictor profiles must be represented in training.
The second condition is a support or positivity condition. If deployment includes profiles absent from training, no weighting of the observed training data can directly reveal the model's conditional performance for those profiles. Even when support overlaps, very large weights can make an estimate unstable; that is a warning about limited effective information, not a reason to assume the estimate is reliable.
The justification follows by expressing deployment risk as an integral and multiplying by a density ratio. Under the conditional-invariance assumption:
Thus the density ratio \(w(x)=p_{\mathrm{dep}}(x)/p_{\mathrm{tr}}(x)\) changes the averaging distribution from training to deployment. It does not repair a changed outcome relationship, missing support, or incompatible measurements. In practice, estimating density ratios also introduces modeling choices; a weighted result should be reported as an estimate under stated assumptions, not as direct evidence that those assumptions hold.
Worked Example: Reweighting Error Under Covariate Shift
Suppose two predictor strata, A and B, have proportions 0.80 and 0.20 in a hypothetical training population, but 0.40 and 0.60 in deployment. Assume the model's classification error is 0.10 in stratum A and 0.40 in stratum B in both populations. This invariance is an assumption for the example; it would require outcome-labeled data or substantive justification in an application.
The training-population error rate is:
The deployment error rate, using the deployment stratum proportions, is:
The model's conditional error rates were stipulated to be unchanged, but the deployment population contains more members of the higher-error stratum. Its expected error therefore rises from 16% to 28%. The stratum weights are \(0.40/0.80=0.50\) for A and \(0.60/0.20=3.00\) for B; applying them to the training contributions gives \(0.80(0.50)(0.10)+0.20(3.00)(0.40)=0.28\). This calculation is justified only under the stated stability and overlap assumptions.
Prevalence Changes and Prediction Values
Outcome prevalence is the proportion of individuals with \(Y=1\). It affects the interpretation of a classifier's errors and correct predictions. If sensitivity \(P(\hat{Y}=1\mid Y=1)\) and specificity \(P(\hat{Y}=0\mid Y=0)\) remain fixed, a prevalence change can still alter positive predictive value \(P(Y=1\mid\hat{Y}=1)\), negative predictive value, and accuracy.
Worked Example: Same Sensitivity and Specificity, Different Prevalence
Consider a hypothetical classifier with sensitivity 0.80 and specificity 0.90. In training, suppose outcome prevalence is 0.20. For 1,000 individuals, the expected counts are 200 with the outcome and 800 without it. The classifier correctly identifies \(0.80(200)=160\) outcome-positive individuals and produces \(0.10(800)=80\) false positives. Its positive predictive value is:
Now suppose deployment prevalence is 0.05, while sensitivity and specificity remain 0.80 and 0.90. Among 1,000 deployment individuals, there are 50 outcome-positive and 950 outcome-negative individuals. Expected true positives are \(0.80(50)=40\); expected false positives are \(0.10(950)=95\). Therefore:
In this hypothetical example, positive predictive value falls from about 66.7% to 29.6%, despite unchanged sensitivity and specificity. Accuracy is \(160/1000+720/1000=0.88\) in training and \(40/1000+855/1000=0.895\) in deployment. The higher deployment accuracy does not mean the classifier is more useful for identifying positive individuals: accuracy is dominated by the larger outcome-negative group. These calculations assume the stated conditional rates hold in deployment; a changed prevalence by itself does not establish that they do.
What Marginal Comparisons Cannot Show
Predictor summaries, plots, and cohort metadata can reveal that populations differ. They cannot, without outcome information, establish whether the mapping from predictors to outcomes remains stable. A domain classifier that distinguishes training from deployment records can indicate differences in observed predictors, but it does not identify the cause of those differences or verify transportability. Conversely, failure to distinguish the cohorts with a particular diagnostic does not prove that all relevant distributions are identical.
Worked Example: Identical Predictor Mix, Changed Conditional Error
Suppose hypothetical training and deployment populations each contain 50% of stratum A and 50% of stratum B. The model's training error is 0.10 in A and 0.20 in B, giving \(0.50(0.10)+0.50(0.20)=0.15\), or 15% overall. In deployment, suppose error is 0.30 in A and 0.20 in B. The deployment error is \(0.50(0.30)+0.50(0.20)=0.25\), or 25%.
The predictor-stratum proportions match exactly, yet overall error differs because conditional performance in A changed. Detecting this requires outcome labels in deployment and comparable outcome ascertainment; the predictor distribution alone cannot reveal the error rates. The example does not identify why conditional error changed. Possible explanations include a genuine change in the outcome relationship, changed measurement procedures, or a difference in how the outcome was defined or recorded.
In multi-omics studies, assay platform, specimen handling, missing-modality patterns, and clinical eligibility criteria are especially important to examine. A shift in a molecular feature might represent a change in patient biology, technical measurement, or both. Batch adjustment and preprocessing must be learned and applied in a way that preserves honest evaluation, as established in the earlier material on fold-local preprocessing and batch adjustment. A score cannot distinguish biological transport failure from a technical mismatch unless the study design and measurements support that distinction.
Diagnosing Shift and Delimiting Claims
A practical assessment begins by naming the intended deployment population and comparing it with the evaluation population. Then inspect predictor distributions and outcome prevalence, using plots and summaries appropriate to the variables. Where deployment outcomes are available, assess conditional errors and calibration across clinically relevant groups or predictor ranges. These are complementary checks: a change in \(P(X)\) suggests a different mix; a change in \(P(Y)\) describes outcome frequency; a change in \(P(Y\mid X)\) directly challenges the conditional stability needed for covariate-shift reweighting.
Use metrics appropriate to the intended decision and report the population to which they apply. If a test cohort is a convenience sample, a score describes that cohort under its sampling and measurement conditions; it does not automatically estimate performance in all future patients. If a cohort is held out by site or time, the result provides evidence about that particular separation, not a guarantee for every deployment setting. Leave-one-cohort-out evaluation, covered in the earlier material on that method, probes cross-cohort transport more directly than a random split within one cohort, but it still cannot represent populations absent from the evaluated cohorts.
When deployment labels are not yet available, predictor comparisons can identify potential mismatch and lack of support, but they cannot establish deployment sensitivity, calibration, or error. A transparent report should distinguish observed facts from assumptions: name the evaluated population and metric; state whether deployment outcomes were observed; describe detected distributional differences; and state any assumptions used for weighting or transport. Do not convert an external score into a causal claim, a universal performance guarantee, or evidence about unobserved populations.
Common Mistakes and Limitations
- Assuming a different predictor mix means the relationship changed. A different \(P(X)\) can alter average risk even when \(P(Y\mid X)\) is stable. Conversely, matching predictor marginals does not prove conditional stability.
- Calling any prevalence difference label shift. Prevalence can change because the predictor mix changed. Label shift specifically imposes stability of \(P(X\mid Y)\), which needs justification.
- Using reweighting beyond its support. If deployment profiles were absent in training, the required density ratio is not available there. Reweighting cannot create evidence for unobserved profiles.
- Interpreting a higher accuracy as uniformly better performance. Accuracy can rise as prevalence falls even while positive predictive value decreases, as the numerical example showed.
- Treating cohort separability as a complete diagnosis. It indicates differences detectable from the chosen inputs, not whether those differences harm prediction or whether the outcome relationship is stable.
Reweighting is most defensible when the deployment target is clearly defined, relevant outcome labels support checking conditional performance, there is good predictor overlap, and the shift assumptions are credible. A direct, well-designed evaluation in the deployment population is preferable when feasible. No adjustment based only on training data can guarantee performance under an arbitrary change in the predictor–outcome relationship.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A model has 10% error in each of two predictor strata. The deployment population has a different proportion of the strata. Under the stated rates, must overall error change? Explain.
- Under covariate shift, what two distributional conditions justify estimating deployment risk by density-ratio weighting?
- In the prevalence example, why did positive predictive value fall even though sensitivity and specificity were unchanged?
- Training and deployment have identical predictor distributions, but deployment outcome labels are unavailable. Can this establish that model performance is unchanged? Why or why not?
- A deployment predictor profile has positive probability but was never represented in training. What does this imply for covariate-shift reweighting?
Solutions. (1) No. If both strata have the same 10% error, any mixture gives \(0.10\) overall error, provided those conditional rates truly remain valid. (2) The conditional outcome distribution must be invariant, \(P_{\mathrm{dep}}(Y\mid X)=P_{\mathrm{tr}}(Y\mid X)\), and deployment must have support within training, so the density ratio is defined. (3) At lower prevalence, false positives from the much larger outcome-negative group form a larger share of all positive predictions. (4) No. Matching predictor marginals does not establish that \(P(Y\mid X)\) is unchanged, and without deployment labels conditional errors or calibration cannot be checked. (5) The support condition fails there; training data cannot identify risk for that profile through reweighting.