Prediction Targets And Deployment Populations

Start With the Decision the Prediction Will Support

A multi-omics model can combine measurements from several assays and still answer the wrong question. A model evaluated on patients whose samples were collected after diagnosis, for example, does not thereby establish that it can predict disease before diagnosis. To make an evaluation meaningful, first specify when the prediction is made, for whom it is made, what it predicts, and where it is intended to be used. Those choices define the claim that an evaluation can support.

A prediction is a statement about an outcome given information available at a specified decision time. It is not, by itself, a claim that changing an omics measurement would change the outcome. The aim here is to define the prediction target and the intended population precisely enough that a model’s measured performance has a clear interpretation.

Definition: The prediction time $t_0$ is the time at which the prediction is intended to be generated. The eligible subjects are the people who meet the stated criteria for receiving that prediction at $t_0$. The outcome $Y$ is the prespecified quantity to be predicted over a stated period or at a stated future time. The deployment population is the population of eligible subjects in the setting where the model is intended to be used, including the relevant distribution of their characteristics, measurements, and outcomes.

Define the Target Before Choosing an Evaluation

The prediction time determines which measurements can legitimately serve as inputs. For each omics modality, state whether its measurement is available at $t_0$, and what the measurement represents. A value obtained from a sample collected after the decision time is not part of the information available for that prediction, even if it is stored in the same data set. For a prospective use, the model must also be feasible with the samples, assays, and turnaround times available at the decision point.

Eligibility should describe the people who would actually be offered a prediction. It might require a particular diagnosis, treatment-decision stage, age range, or set of clinical criteria. Eligibility is not simply the set of people who happened to have complete data in a research study. If the intended users are newly diagnosed patients before treatment selection, an evaluation limited to people who completed treatment answers a different question.

The outcome needs an operational definition: what event or quantity is recorded, how it is measured, and when it is assessed. For a binary outcome, specify what counts as an event. For a continuous outcome, specify the measurement and its units. If the target is an event within a horizon $h$, write it as $Y(t_0,t_0+h)$, and explain how events before $t_0+h$, competing events, and incomplete follow-up are handled. These decisions change which observations count as outcomes and therefore change the prediction problem.

The deployment population is more than a hospital name or a country. It includes the distribution of factors that can affect model performance: disease severity, demographic and clinical characteristics, assay platforms, care pathways, and outcome prevalence. The evaluation sample may differ from that population. A useful evaluation must either resemble the intended deployment population or make a justified adjustment for the differences that matter to the performance quantity being reported.

Definition: Let $E=1$ indicate that a subject meets the eligibility criteria, let $X(t_0)$ denote the information available at prediction time, and let $Y$ be the defined outcome. For a model $f$ and a loss function $L$, its target risk in deployment population $D$ is $$ R_D(f)=\mathbb{E}_D\left[L\{Y,f(X(t_0))\}\mid E=1\right]. $$ The loss function specifies what counts as prediction error. The target risk describes average error for eligible subjects in the stated deployment population, not for an unspecified collection of samples.

This expression assumes that the outcome and inputs are defined consistently, the model’s inputs can be obtained at $t_0$, and the evaluation can estimate or credibly represent the target population’s joint distribution of inputs and outcomes. If these conditions do not hold, the reported performance may describe a different prediction task or population. The loss also matters: average squared error, absolute error, and classification error quantify different aspects of performance, so the choice should suit the outcome and intended use. Metric selection is not a substitute for defining the target.

Why Population Alignment Changes Performance

A short decomposition shows why the intended population belongs in the estimand. Suppose eligible subjects fall into subgroups indexed by $z$, such as lower- and higher-risk clinical groups. Write $R_z(f)$ for the model’s mean loss in subgroup $z$, and $w_{D,z}=P_D(Z=z\mid E=1)$ for that subgroup’s proportion among eligible deployment subjects. Then the overall deployment risk is

R_D(f)=\sum_z w_{D,z}R_z(f).

This follows by partitioning the expectation in the definition of $R_D(f)$ across the mutually exclusive subgroups: each subgroup contributes its conditional mean loss multiplied by its probability in the deployment population. If an evaluation instead averages subgroup losses using different proportions, it estimates a different weighted average. It estimates deployment risk only if the subgroup proportions match, or if the analysis uses appropriate weights and the subgroup-specific losses are applicable to deployment. This does not guarantee transportability: performance can also differ within subgroups if measurement processes, care, or other relevant features change.

Worked Example: A Validation Sample With Different Risk Mix

Suppose a hypothetical classifier will be used among eligible patients in a clinic where 20% are in a higher-risk group and 80% are in a lower-risk group. In an evaluation sample, the two groups are equally represented. The classifier’s observed classification error is 30% in the higher-risk group and 10% in the lower-risk group. Assume for this calculation that those subgroup error rates apply to the clinic population.

Unweighted evaluation error. Because each subgroup makes up half of the evaluation sample, its unweighted error is $0.50(0.30)+0.50(0.10)=0.15+0.05=0.20$, or 20%.

Deployment error. The clinic’s subgroup proportions give $R_D(f)=0.20(0.30)+0.80(0.10)=0.06+0.08=0.14$, or 14%.

The 20% figure describes the equally weighted evaluation mix; the 14% figure is the estimated error for the clinic’s stated mix under the assumption that subgroup-specific error rates transfer. They differ by 6 percentage points. Neither number is inherently the right summary for every use: the intended population determines which weighting is relevant. The calculation does not justify transferring the subgroup error rates if the clinic’s assays or care pathways differ in ways that affect performance.

Worked Target Specifications

Worked Example: Predicting an Event After a Baseline Visit

Consider a hypothetical model intended to estimate whether a patient will receive a new diagnosis of a specified disease within two years. The clinical decision is whether to offer closer monitoring at a scheduled visit.

Prediction time. Set $t_0$ as the scheduled visit before the monitoring decision. The candidate inputs are the clinical measurements and omics assays available from samples collected by that visit. Any assay result obtained later is outside the stated prediction-time information.

Eligibility. Include patients attending that visit who meet the clinical criteria for monitoring and do not already have the specified diagnosis at $t_0$. This matches the group for whom the prediction could guide the stated decision.

Outcome and horizon. Define $Y=1$ if the patient receives the specified diagnosis during the two years after $t_0$, and $Y=0$ otherwise, subject to a prespecified rule for incomplete follow-up. The target is a two-year event, not lifetime disease risk or the presence of disease at the visit.

Deployment population and evaluation implication. Suppose the intended use is in eligible patients attending a network of community clinics. An evaluation in a specialist clinic could still inform model performance, but it does not directly establish performance in the network if the patients, assays, or diagnosis processes differ. A report should name the population represented and avoid presenting a specialist-clinic result as a direct estimate for community deployment without a defensible transport argument.

Worked Example: Separating Two Plausible Outcomes

A hypothetical oncology team wants a prediction at the treatment-decision visit for eligible patients about to begin a specified therapy. At 12 months after that visit, records classify 100 patients into three mutually exclusive categories: 20 have documented disease progression, 10 have died without documented progression, and 70 are alive without progression.

If the intended target is the composite “progression or death by 12 months,” the event count is $20+10=30$. The observed event proportion is $30/100=0.30$, or 30%.

If the intended target is “documented progression by 12 months,” the count is 20 and the observed proportion is $20/100=0.20$, or 20%. Those who died without documented progression are not automatically equivalent to patients who remained alive and progression-free. A rule for deaths and other incomplete outcomes must match the chosen endpoint and be specified before evaluation.

Thus the same eligible group and prediction time can yield different prediction targets. The 30% result is not a correction to the 20% result; it summarizes a different outcome. A model intended to support a decision about progression-or-death should be evaluated against that composite, not against progression alone.

Evaluation Designs Must Match the Intended Use

Once the target is specified, ask whether the evaluation reproduces its essential features. An evaluation in a similar setting and time period can provide evidence about performance for that setting. A later cohort can be informative when the intended use is future deployment, provided eligibility, prediction time, outcome definition, and measurement are aligned. An evaluation in a different institution or population addresses transport to that setting; its relevance depends on what differs and whether those differences alter the model’s errors.

These designs answer distinct questions. An internal evaluation can assess performance within the source population, but a close match to that population does not by itself show how the model will perform in a new health system. A temporal evaluation can examine performance in a later period at the same setting, but it may not reveal differences between institutions. An external evaluation can examine another setting, but a single external site does not establish performance in all possible deployment populations. No design can compensate for an outcome that does not match the intended target or inputs that would not be available at prediction time.

The earlier material on calibration and discrimination distinguishes useful performance summaries. Here the key point is that each summary is conditional on the target. For instance, event prevalence influences quantities such as positive predictive value, while average loss can change when subgroup proportions change. Report the population and eligibility criteria alongside any metric, and be cautious when comparing numbers calculated for different targets or populations.

State the decision time.
Define $t_0$ and list which clinical and omics measurements are available by that time.

Name the eligible subjects.
Use criteria that identify the people who would actually receive the prediction for the stated decision.

Specify the outcome.
Define the event or quantity, its measurement, the follow-up horizon, and rules for competing or incomplete outcomes.

Describe deployment.
Identify the intended setting and relevant population characteristics, measurement processes, and care context.

Align the evaluation claim.
Check that the evaluation represents the target inputs, subjects, outcome, and population; state any remaining differences and assumptions.

Diagnostics, Pitfalls, and Limits

Several warning signs indicate that the evaluation may not answer the intended question. The cohort may include people who would not be eligible in deployment; the stated prediction time may precede some input measurements; the endpoint may use a different horizon or event definition; or the evaluation setting may have a substantially different case mix or measurement process. These are not minor reporting details. Each can change either the prediction target or the distribution over which performance is averaged.

“The data set has these patients, so they are the target population.” This reverses the logic. Define intended users from the decision context, then assess whether the data represent them.
“The outcome is recorded, so its meaning is obvious.” A diagnosis, a composite event, and a measurement at a fixed time are different targets. Write down the event and horizon.
“The model performed well at one hospital, so it will work everywhere.” A result supports a claim about the evaluated population; generalization to another setting needs evidence and assumptions about differences between settings.
“The same metric means the same target.” A shared metric name does not make two evaluations comparable if their populations, eligibility criteria, outcomes, or horizons differ.

Target specification cannot eliminate uncertainty. Some intended populations may be poorly represented in available data, outcomes may be measured differently across sites, and future clinical practice may change. Weighting can address a known difference in subgroup proportions only when subgroup-specific performance is adequately estimated and relevant to deployment; it cannot repair absent evidence for a group or an outcome measured incompatibly. These limits should narrow the conclusion rather than be hidden by a single aggregate score.

A useful final check is to complete this sentence: “Among [eligible subjects] in [deployment setting], at [prediction time], using [information available then], the model predicts [outcome] over [horizon], and performance is summarized by [metric].” If any bracket cannot be filled precisely, the evaluation claim is not yet well defined.

Check Your Understanding

Answer each question by tying the design choice to the prediction claim. Brief solutions follow.

A model is intended to guide a decision at diagnosis, but its evaluation uses omics assays collected six months after diagnosis. What is the mismatch?
A validation sample is 50% high-risk and 50% low-risk. The deployment population is 20% high-risk and 80% low-risk. What additional information is needed to estimate deployment classification error by standardization?
Why must “death or progression within one year” and “progression within one year” be treated as distinct outcomes?
What can an evaluation in one specialist hospital establish, and what can it not establish about deployment in community clinics?

Solutions. (1) The evaluation uses measurements unavailable at the intended prediction time, so it does not directly evaluate the stated decision-time prediction task. (2) The error rate in each subgroup and a justification that those subgroup rates apply in deployment; the standardized error is $0.20R_{\text{high}}+0.80R_{\text{low}}$. (3) They count different events: death without progression is included in the composite but not necessarily in the progression-only endpoint, so outcome labels and performance can differ. (4) It can provide evidence for eligible patients in that specialist-hospital setting. It does not, by itself, establish performance for community-clinic patients, whose case mix, assays, and care context may differ.