Defining the Genomic Prediction Target

Why the Prediction Target Comes First

A genomic prediction model can perform well for one intended use and poorly for another. A model evaluated on held-out individuals from the same cohort may be useful for predicting additional individuals recruited under similar conditions. That result does not automatically tell us how well the model will predict patients at a different hospital, in a later period, or under a different assay protocol. The populations and prediction circumstances differ, so the performance target differs.

The prediction target is not merely the name of an outcome. It specifies who or what will receive predictions, when predictions are made, what information is available then, and how prediction quality will be measured. In genomic applications, these choices matter because populations, clinical practice, genotyping platforms, and outcome definitions can vary.

Risk Depends on the Target Distribution

Let \(X\) denote the predictors available at the time of prediction, including genomic measurements when appropriate, and let \(Y\) be the outcome defined for the prediction task. Let \(f\) be a fitted prediction rule and \(L(Y,f(X))\) a loss, where smaller values mean better predictions. For example, squared-error loss is \(L(Y,f(X))=(Y-f(X))^2\); misclassification loss is 1 when the predicted class is wrong and 0 otherwise.

For a fixed fitted rule, its target risk is the expected loss for prediction occasions drawn from the target population. The distribution \(P^*\) must describe the population and circumstances specified by the prediction target.

The risk is conditional on the fitted rule \(f\). If the goal is instead to describe the performance of a training procedure, which can produce different fitted rules from different training samples \(D\), the target may average over those training samples as well:

where \(A\) is the specified procedure that maps training data to a fitted prediction rule. These are different quantities: one concerns a particular fitted model, and the other concerns the expected performance of the procedure. Either can be relevant, but the intended quantity should be stated.

When Does Validation Estimate the Intended Risk?

Suppose a fitted rule \(f\) is fixed before evaluation, and a validation sample \((X_i,Y_i)\), \(i=1,\ldots,m\), consists of independent draws from \(P^*\). If the loss has a finite expectation, the average validation loss estimates the target risk:

The justification is direct. Each validation loss has expectation \(R(P^*,f)\), so linearity of expectation gives:

This result does not say that every validation sample will produce the exact risk. The estimate varies from sample to sample. It also depends on the assumptions: the validation observations must represent the target distribution, the fitted rule must not have been adapted using their outcomes, and the recorded predictors and outcomes must correspond to the stated prediction task. If validation observations are selected in a way that changes their distribution, an unweighted average generally estimates performance in that selected sample rather than in \(P^*\).

Same Cohort and New Site Are Different Targets

The phrase “generalize to new patients” is incomplete unless it says what those patients are like. A same-cohort target treats future patients as drawn from the cohort's population of prediction occasions. A new-site target concerns patients from a site not represented in the same way during model fitting or evaluation. Differences between sites can include patient mix, clinical practice, outcome measurement, and genomic assay procedures. These differences can change the distribution of prediction errors.

For a target that draws sites and then patients within sites, overall risk can be written as an average of site-specific risks. If site \(s\) has target weight \(w_s\), with nonnegative weights summing to one, then:

where \(R_s(f)\) is the expected loss for a patient at site \(s\). The weights encode the target: for example, they could reflect the proportions of patients expected from each site, or assign equal importance to each site. Those choices need not produce the same risk. Data from one site cannot, on their own, establish performance for a population of future sites.

Population Composition and the Performance Measure

Even when the model, outcome, and loss definition are unchanged, changing the mix of individuals can change average performance. This is especially important for classification. The fraction of individuals with each outcome class affects overall misclassification error, and measures such as sensitivity and specificity answer different questions from overall error. The measure must therefore be chosen to match the intended use, rather than selected after seeing which one looks most favorable.

State the Target Before Choosing an Evaluation Design

A useful target statement is concrete enough that another analyst could determine whether a candidate validation sample is relevant. It should identify:

• Prediction unit and population: for example, patients newly presenting to specified clinics, rather than samples or repeated records treated as independent people.
• Prediction time and available information: predictors must be available when the prediction would actually be made. A variable recorded only after the outcome is known is not a valid predictor for that use.
• Outcome and horizon: define what is being predicted and when or how it is ascertained.
• Performance measure: specify the loss or summary, including any clinically meaningful weighting.
• Population mixture: state whether patients, sites, or subgroups receive weights proportional to their expected frequency or by another justified rule.

These statements distinguish targets that are easy to blur together. For instance, “predict disease status using genomic data” leaves out the intended patients, the time of prediction, the disease definition, and the measure of success. A more useful description might specify newly presenting patients at a defined set of clinics, genomic and clinical predictors measured at the initial visit, disease status assessed at a stated follow-up, and patient-weighted misclassification loss.

Diagnostics, Alternatives, and Limitations

Before interpreting validation performance, compare the validation sample with the target description. Check whether relevant populations, sites, calendar periods, outcome definitions, and measurement procedures match. Examine whether the validation set includes the kinds of individuals expected at deployment and whether important subgroups are represented. Large discrepancies in predictor distributions or outcome frequencies are warnings that an unweighted validation average may not estimate the stated target risk. Similar observed distributions are reassuring but do not prove that all relevant differences have been ruled out.

There is no universally best evaluation population. A random held-out sample from the same cohort is a natural choice for a same-cohort target. Evaluation at a separate site is more directly relevant to a target involving that site or similar sites, but a single external site may not represent all future sites. A time-separated evaluation can be relevant when the intended use is in a later period, while changes in practice or measurement can still limit transfer. These are alternatives for answering different questions, not interchangeable tests of model quality.

If the validation sample and target population differ, weighting can sometimes align the average with a specified target mixture when subgroup risks can be estimated and the needed groups are observed. It cannot supply evidence for populations absent from the data, guarantee that conditional prediction performance is unchanged, or correct a changed outcome definition. Transporting a performance estimate therefore requires assumptions about how the validation and target populations relate, as well as adequate data to assess those assumptions.

A common failure is to report a single score as “the model's accuracy” without saying which population it describes. Another is to define the target broadly as “future patients” while evaluating only one cohort, or to count multiple records from the same individual as if they represented independent future individuals. The correction is to define the prediction occasions and target distribution first, then ask whether the evaluation data and its analysis support an estimate for that target. The exact validation split or resampling unit is a separate design decision; it cannot repair an unclear target.