Prediction Targets and Validation Populations

Why a Prediction Target Must Be Specified

A polygenic score is not a prediction target by itself. It is an input to a prediction rule. To evaluate that rule, one must first decide exactly what outcome is predicted, when the prediction is made, which individuals the prediction is intended to describe, and what counts as one prediction. These choices determine what data can validate the rule and what a reported performance summary means.

For example, a score might be used to predict whether an adult without a previous diagnosis will develop a condition within five years. That is a different task from predicting whether a newborn will receive the diagnosis by age five, or estimating a quantitative trait value at a routine visit. The underlying score could be identical, but the eligible individuals, outcome records, time window, and relevant population would differ.

Outcome, Prediction Time, and Horizon

The outcome must be defined in a way that supports consistent measurement. It may be a binary diagnosis, a quantitative measurement, or another specified endpoint. A definition should make clear how the outcome is ascertained, which events count, and how missing or ambiguous records are handled. Different phenotype definitions can produce different prediction targets even when they use the same broad disease label.

The prediction time is the point at which the prediction would be issued. Predictors must be available by that time. For a prospective prediction, the outcome window generally begins after the prediction time; information recorded later cannot be treated as a predictor available at prediction time. The time horizon says how far ahead the outcome is assessed. For instance, “diagnosis within five years after enrollment” specifies a horizon, whereas “diagnosis by age 50” uses age as the endpoint.

For a time-indexed outcome, write \(Y_i(\tau)\) for the outcome defined for individual \(i\) at horizon \(\tau\), and \(X_i(t_0)\) for information available at prediction time \(t_0\). A prediction rule \(f\) maps that information to a prediction, \(f\{X_i(t_0)\}\). The notation emphasizes that changing \(t_0\), \(\tau\), or the outcome definition changes the task. For a binary five-year endpoint, for example, \(Y_i(5)=1\) might mean that an eligible individual develops the defined condition within five years after \(t_0\).

Target Population, Unit, and Performance Estimand

The target population is the group for whom the prediction rule is intended to be used. It should be defined by eligibility, setting, time period, and other features relevant to intended use. In genetic prediction, population characteristics that affect genotype distributions, outcome frequencies, or predictor-outcome relationships can matter. A broad label alone may not fully describe the intended population.

The unit of prediction specifies what receives one prediction and therefore what receives weight in a performance summary. Often it is one individual. It might instead be a pregnancy, family, clinical visit, or another unit, depending on the intended use. Repeated observations from one person do not automatically represent multiple independent people or multiple equally important prediction targets.

For a prediction rule \(f\), a general performance estimand can be written as the target-population expectation of a loss:

Here \(Y\) is the target outcome, \(X\) is the information available at the specified prediction time, and \(L\) measures the discrepancy between the outcome and prediction. The expectation is taken over the target population and its distribution of outcomes and predictors. The particular loss must suit the prediction task. For a binary outcome, one possible loss is squared error between the outcome coded 0 or 1 and a predicted probability. This expression is a general framework, not a claim that one metric is appropriate for every task.

The validation sample can estimate \(\Psi\) directly when its observations represent the target population and the outcome and prediction definitions match. To see why, suppose \(n\) validation units are independently sampled from the target population and their losses are \(L_1,\ldots,L_n\). The sample mean loss is

Because each \(L_i\) has expectation \(\Psi\), linearity of expectation gives

Thus the sample mean loss targets the stated population mean under this sampling condition. If the validation sample has a different composition, its unweighted mean generally describes that sample’s population instead. A different sampling design may require a corresponding analysis to target the intended population; it does not make the two populations interchangeable.

Matching the Validation Population to the Target

A validation sample is suitable only relative to a stated prediction target. At a minimum, check that sample members are eligible for the target use, that predictors are available at the correct prediction time, that outcomes are measured with the intended definition and horizon, and that the sampling structure corresponds to the prediction unit. Then ask whether the sample’s distribution represents the population for which performance is being claimed.

If the sample differs from the target population, a performance estimate can still be useful, but its interpretation must be limited to the population it represents unless a justified method connects it to the target. For example, an estimate from a specialty clinic may describe performance among clinic attendees rather than all people eligible for population screening. A sample with a different mixture of age groups or sites may likewise produce an overall average that does not apply to the intended mixture.

The difference is visible even when group-specific performance is measured consistently. Suppose a hypothetical validation sample has 20 people in each of two groups. The sum of individual squared-error losses is 1.6 in group A and 6.4 in group B. The group mean losses are \(1.6/20=0.08\) and \(6.4/20=0.32\), respectively. The sample gives the groups equal weight, so its overall mean loss is

Now suppose the intended target population is 75% group A and 25% group B, and the group-specific mean losses are applicable within those groups. The target-weighted mean loss is

The values 0.20 and 0.14 are not contradictory: they average the same group-specific quantities using different group proportions. This calculation would be justified only if the group definitions and loss are the same in both settings, and the validation group means can reasonably represent the corresponding target groups.

A Practical Specification Sequence

Before collecting or analyzing validation data, write a concise target specification. The following sequence exposes decisions that otherwise remain hidden.

Common Mistakes and Why the Specification Matters

• Calling a broad disease label the outcome definition. Two studies may use different diagnostic rules or ascertainment windows. State the operational definition and the time period rather than relying only on a disease name.
• Leaving the prediction time implicit. A predictor measured after the point when a real prediction would be made cannot support that intended use. Identify what is available at the stated prediction time.
• Treating follow-up length as interchangeable. No diagnosis by one year does not establish no diagnosis by five years. The horizon determines which outcome is being predicted.
• Reporting sample performance as universal. A statistic from a selected clinical sample describes the population represented by that sample unless further assumptions and methods support broader transport.
• Counting repeated records as independent target units by default. Several visits from one person can change the weighting and the estimand. Decide whether the target is a person, visit, or another unit before averaging.

These specifications are especially important for genetic prediction because a score can be fixed while its intended use changes across settings, ages, outcome definitions, and prediction horizons. “How well does this score predict?” is incomplete until the outcome, timing, population, and unit are named. A precisely defined target makes both the validation sample and the resulting performance claim interpretable.