Defining Transition-Specific Validation Targets

Why a Validation Target Must Be Defined First

A prediction cannot be judged as accurate or inaccurate without first specifying what it is predicting and for whom. In a multi-state setting, “risk of disease progression” may refer to several different quantities: a direct transition from one state to another, being in a destination state at a later time, or an instantaneous transition rate. These quantities answer different questions. They can differ even when they concern the same patients and the same pair of states.

A transition-specific validation target therefore needs more than a transition label. It must identify the population under consideration, the state occupied at the time a prediction is made, the destination of interest, the prediction time, and the length of time over which the outcome is defined. It must also say how exits to other states are treated. These choices give the predicted quantity its meaning before any assessment of prediction performance is attempted.

Definition: A transition-specific prediction target is a precisely defined outcome probability for a specified target population, conditional on a specified origin state and information available at a prediction time, concerning a specified destination transition over a specified horizon. A complete target states whether it concerns a direct transition or occupancy of a state at a later time, and how other possible transitions enter its definition.

Specify the Target in Five Parts

Let \(X(t)\) denote the state occupied at time \(t\), and let \(s\) be the prediction time. Write \(i\) for an origin state and \(j\) for a distinct destination state. Let \(h>0\) be the prediction horizon, so the target interval runs from \(s\) to \(s+h\). Let \(Z(s)\) denote the covariates or history available for prediction at \(s\). A transition target is usually conditional on being eligible at \(s\), in particular on occupying state \(i\) then.

Target population. State which individuals the target describes. For example, it may be all eligible patients in a named care setting, or a specified subgroup of them. The population also determines the distribution of covariates and histories over which any population-average target is defined.
Origin state. State the condition at prediction time, such as \(X(s)=i\). This makes clear who is at risk for the transition being predicted. It is not generally the same as saying that everyone belonged to state \(i\) at baseline.
Destination and event definition. Name the transition of interest and say whether the outcome is a direct \(i\to j\) transition or being in state \(j\) at the end of the horizon. The distinction matters if paths through other states are possible.
Prediction time. Specify when the prediction is made and what information is available then. The prediction must not use later information. If predictions are made at a landmark time \(s\), the target population consists of individuals eligible at that landmark, not automatically the original baseline cohort.
Horizon. Specify how far ahead the outcome is assessed. “Within one year after prediction” and “by age 70” describe different time intervals unless the prediction time makes them coincide.

A useful formal target for a direct transition is the conditional probability

r_{ij}(s,h\mid Z(s)) = P\!\left(\text{the next transition after }s\text{ is }i\to j \text{ and occurs by }s+h \mid X(s)=i,\ Z(s)\right).

This definition treats another outgoing transition from \(i\) before \(i\to j\) as a competing outcome for this direct-transition target. Remaining in \(i\) through \(s+h\) also means that the specified transition did not occur within the horizon. The definition is not a claim that every individual must eventually make a transition.

If the intended outcome is instead occupancy of state \(j\) at the horizon, the event is \(X(s+h)=j\), conditional on \(X(s)=i\) and the specified information. That target can include paths through intermediate states. It is not interchangeable with the probability of a direct \(i\to j\) transition. In settings where a transition can recur, the target must additionally state whether it concerns the first such transition or any occurrence within the interval.

From Individual Probabilities to a Population Target

A model may produce a probability conditional on a person’s information, while a validation question may concern an average in a population. These are different levels of target. For a target population distribution of \(Z(s)\), a population-average direct-transition probability is the mean of the conditional probabilities across that distribution:

\theta_{ij}(s,h) = E_{\mathcal{P}}\!\left[ r_{ij}(s,h\mid Z(s)) \mid X(s)=i \right],

where \(\mathcal{P}\) denotes the stated target population. The conditioning indicates that the average is over members of that population who occupy the origin state at prediction time. If the population or its distribution of prediction-time information changes, this average can change even when the conditional prediction rule is unchanged. Thus “average risk” is incomplete unless its target population is named.

The probability target also depends on a coherent event definition and reliable state ascertainment. The states and transitions must represent the clinical question, eligibility at \(s\) must be defined, and the information used for prediction must be available at \(s\). A validation sample should represent the intended target population, or differences must be addressed by an appropriate design or analysis. These requirements define the question; they do not by themselves determine how to estimate its answer under incomplete follow-up.

Worked Example: Direct Progression Before a Competing Exit

Hypothetical setting. A clinic uses a model to predict disease progression. State 1 is alive without progression, state 2 is alive after progression, and state 3 is dead. Predictions are issued at a scheduled review. Define a target for progression within 12 months.

Population and origin. The target population is patients attending that review who are alive and progression-free at the review. Thus the origin is state 1 at prediction time \(s\), not state 1 at initial diagnosis.

Destination, time, and horizon. The destination is state 2, the prediction time is the review date, and \(h=12\) months. The target is the probability of a direct \(1\to2\) transition within those 12 months. Death before progression is a competing exit from state 1, and a patient who remains progression-free through month 12 has not had the target event.

Suppose, purely for illustration, the target population has two equally sized groups with conditional target probabilities of 0.20 and 0.50. The population-average probability is

\theta_{12} = 0.50(0.20)+0.50(0.50) = 0.10+0.25 = 0.35.

The value 0.35 is the average probability of the defined direct transition in this hypothetical population, or 35.0% over 12 months. It is not the probability of being in state 2 at month 12: a patient might progress and later die, and that patient would not occupy state 2 at month 12. Nor does the average apply automatically to all patients at diagnosis, since the target population is restricted to those eligible at the review.

A Short Justification for Population Averaging

The averaging formula follows from the law of total expectation. Let \(Y_{ij}(s,h)\) be the indicator that the defined direct transition occurs within the horizon. Within the eligible population, its conditional expectation given prediction-time information is the conditional event probability:

E\!\left[Y_{ij}(s,h)\mid Z(s),X(s)=i\right] = r_{ij}(s,h\mid Z(s)).

Taking expectations over the target population’s distribution of \(Z(s)\), conditional on occupying state \(i\), gives

E\!\left[Y_{ij}(s,h)\mid X(s)=i\right] = E_{\mathcal{P}}\!\left[ r_{ij}(s,h\mid Z(s))\mid X(s)=i \right] = \theta_{ij}(s,h).

This identity explains why a population-average target depends on who is averaged over. Changing the target population’s mix of covariates or histories can change \(\theta_{ij}(s,h)\), even if the conditional probabilities for each covariate pattern remain the same.

Worked Example: Distinguishing a Direct Transition From Occupancy

Hypothetical setting. Consider an illness-death model with state 1 for alive without illness, state 2 for alive with illness, and state 3 for dead. At a prediction visit, a patient is in state 1. The question concerns the next 24 months.

If the target is a direct \(1\to2\) transition within 24 months, a path \(1\to3\) does not count as the event, and a path \(1\to2\to3\) does count because the direct progression occurred before death. The outcome is whether the first transition out of state 1 is to state 2 by month 24.

If the target is instead \(X(s+24)=2\), only patients who are alive with illness at month 24 count. A path \(1\to2\to3\) does not satisfy that endpoint, while a path through another state that ends in state 2 could satisfy it if such paths are part of the model. These outcomes differ because one concerns a transition during the interval and the other concerns the state at its endpoint.

For an arithmetic illustration, suppose a hypothetical set of 100 predictions includes 30 people whose path is \(1\to2\to3\) before month 24, 20 whose path is \(1\to2\) and who remain in state 2 at month 24, and 50 who do not enter state 2 during the interval. The direct-transition event occurs for \(30+20=50\) people, or \(50/100=0.500\). Endpoint occupancy of state 2 occurs for 20 people, or \(20/100=0.200\). The arithmetic illustrates distinct event definitions; it does not estimate a population probability from a validation study.

Worked Example: A Prediction Made at a Later Landmark

Hypothetical setting. A transplant program predicts graft failure. State 1 is alive with a functioning graft, state 2 is graft failure, and state 3 is death with a functioning graft. The model is used at the 6-month visit to predict the next 12 months.

The target population is patients who are alive with a functioning graft at month 6 and have the information required by the prediction rule at that visit. The origin is state 1 at \(s=6\) months; the destination is state 2; and the horizon is \(h=12\) months, ending at month 18. The direct-transition target is the probability of graft failure from state 1 by month 18, before death while the graft is functioning.

Patients whose graft failed or who died before the month-6 prediction time are not members of this landmark target population. The target does not describe all transplant recipients from surgery onward. If the model uses current laboratory values at month 6, those values belong to \(Z(6)\); measurements made after month 6 cannot be used to define the prediction information.

Suppose the hypothetical conditional probabilities are 0.10 for 60% of eligible patients and 0.25 for 40%. The population average in this specified landmark population is

\theta = 0.60(0.10)+0.40(0.25) = 0.06+0.10 = 0.16.

This is 16.0% over the 12 months following the 6-month visit. It is not a 16.0% risk from the time of transplantation, nor a risk for patients who were no longer in state 1 at month 6.

Assumptions, Diagnostics, and Common Misinterpretations

A target is interpretable only if its components can be applied consistently. At minimum, the state definitions must be mutually understandable in the setting, the origin state must be observable at prediction time, and the destination event must be distinguishable from other exits. The time scale and time units must be explicit. If prediction-time history is used, its measurement window and availability must be clear. These are substantive and data-quality conditions, not merely notation.

Several practical checks can reveal a poorly specified target:

Eligibility check: Can each person be classified as eligible or ineligible at prediction time using information available then?
Event check: Can the defined \(i\to j\) event be distinguished from competing exits and from being in state \(j\) at a later time?
Time check: Do the prediction time and horizon determine one unambiguous interval in the stated units?
Population check: Does the described population match the people for whom predictions will be used and assessed?
Information check: Are all predictors available at \(s\), rather than measured after the prediction time?

A further issue is support: the validation data need to contain people and prediction-time histories relevant to the stated target population. If important subgroups or transitions are absent, conclusions about them are not justified merely because the model returns a number for them. Likewise, incomplete or inconsistent state ascertainment can prevent the outcome from being classified according to the target. The target remains conceptually definable, but the available data may not identify it without additional assumptions or methods.

The most common mistake is to use “transition probability,” “risk,” “hazard,” and “state probability” as though they meant the same thing. A transition hazard is an instantaneous rate among individuals still in the origin state; it is not itself the probability of making that transition by a horizon. A direct-transition probability concerns a specified transition during an interval. A state-occupancy probability concerns the state at a particular time and can include paths through other states. An overall event risk may combine destinations and so fail to isolate the transition of interest.

The correction is to write the event in words before naming a statistic: who is eligible, what state they occupy when the prediction is issued, what event counts, when it must occur, and what other transitions mean for that event. This precision is also essential when comparing predictions from different models: models aimed at different populations, landmarks, or event definitions do not target the same quantity, even if both output numbers between zero and one.

Key takeaway: A transition-specific validation target is not defined by the edge \(i\to j\) alone. It requires a target population, origin state at prediction time, event definition, prediction time, and horizon. State whether the target is a direct transition or endpoint occupancy, and state how competing exits are handled.

Check Your Understanding

For each question, give a precise answer. Brief solutions follow the questions.

A model predicts a direct transition from state A to state B within six months for people in state A at a clinic visit. Name four additional details needed to make the target fully interpretable.
In an illness-death model, explain why “transition from illness-free to illness within one year” and “alive with illness at one year” need not describe the same event.
A prediction is made at month 12 for people still in the origin state then. Why should this target not automatically be described as applying to the original baseline cohort?
For a direct transition target, a person leaves the origin state for a competing destination before reaching the destination of interest. Does that person have the specified direct transition within the horizon? Explain.
Two groups have conditional transition probabilities 0.10 and 0.30. If their proportions in the target population are 0.25 and 0.75, calculate the population-average probability.

Solutions. (1) Specify the target population, the exact origin and destination states, the prediction time or visit definition, the six-month interval’s time origin, and the treatment of competing exits; also state the prediction information available at the visit. (2) A person may enter the illness state and then die before the one-year endpoint. That person had the direct transition but is not alive with illness at one year. (3) People who left the origin state or died before month 12 are not eligible for a prediction at that landmark. The month-12 target is conditional on eligibility then and therefore concerns a selected landmark population. (4) No. The competing exit occurs before the specified direct transition, so the defined direct-transition event has not occurred. (5) The average is \(0.25(0.10)+0.75(0.30)=0.025+0.225=0.250\), or 25.0% in the stated target population.