Validation After Entry Into Intermediate States

Why Entry Into an Intermediate State Changes the Validation Question

A prediction issued when a person enters an intermediate state answers a conditional question: among people who have reached this state and are eligible for the prediction, what happens next? The relevant validation sample is therefore not automatically the full cohort that began in the initial state. It is the set of eligible state entrants to whom the prediction could actually have been issued.

For example, a model might predict the probability of transition from a post-surgical state to readmission within 30 days. A person who never enters the post-surgical state cannot receive that particular prediction. Including such a person as a non-readmission in the validation data would change the question and distort the evaluation. Conversely, including only entrants who remain in the state for some additional period after entry would select on future information.

The key design decisions are made before calculating a score: define the eligible entry event, the instant at which prediction is available, the outcome window, the destinations counted as competing outcomes, and how censoring is handled. These choices determine what population and prediction task the validation results describe.

Define Eligibility and the Prediction Time

Let \(r\) be the intermediate state of interest, and let \(T_i\) denote the time individual \(i\) enters \(r\). Suppose the prediction is issued at that entry. For a horizon \(h\), the prediction of a transition to destination \(k\) is the model's probability of reaching \(k\) by \(T_i+h\), conditional on the information available at \(T_i\). Write this prediction as \(p_{ik}(h)\). Its information set may include the path and covariates observed before or at entry, but not subsequent events or measurements.

Eligibility must be operational rather than implicit. Specify the entry definition, the population and calendar period from which entries are sampled, any minimum data requirements, and which prediction version is evaluated. If a person can enter state \(r\) repeatedly, decide whether the unit is the first eligible entry or every eligible episode. Using every episode gives more weight to people with more entries unless the analysis explicitly targets episodes; episodes from the same person are also not independent for uncertainty calculations.

Do not require entrants to survive, remain in state \(r\), or stay uncensored for a period after \(T_i\) just to qualify. Those conditions use information from after prediction time. An entrant who transitions to a competing destination shortly afterward is still eligible: that transition is an observed outcome, not a reason to remove the episode. An entrant censored before \(T_i+h\) is likewise eligible, although the outcome may need censoring adjustment.

Make Follow-Up Comparable Across Entry Times

Entrants can arrive at different calendar times. To evaluate a fixed prediction horizon, define each outcome window relative to that person's prediction time: for a 30-day prediction issued at \(T_i\), assess events through \(T_i+30\) days. Do not instead assess everyone only through one shared calendar date if that gives early entrants 30 days of observation and late entrants much less.

A fixed calendar cutoff can still be part of the data-collection design. It determines which entrants have complete follow-up and which are censored before the prediction horizon. For example, an entry on June 28 in a database ending June 30 has only two days of potential observation for a 30-day target. It is not valid to label this person event-free at day 30. The episode can be retained with an appropriate censoring method, or excluded under a clearly defined complete-follow-up rule; the choice changes the target sample and should be reported.

Entry timing can also matter because entrants in different calendar periods may differ in treatment, clinical practice, or case mix. If the target is the actual mix of eligible entrants during the stated period, the validation summary naturally reflects that mix. If the intended target has a different entry-time distribution, the validation data may need prespecified standardization or sampling weights. A single pooled result should not be assumed to apply unchanged to every calendar period.

The prediction horizon in this tutorial is defined from issuance so that its meaning is unambiguous for validation. This operational definition does not by itself determine how the prediction model represents time spent in a state; that is a separate modeling question.

Assign Outcomes Under Competing Transitions

For a transition-specific evaluation, define the outcome for every eligible entrant using the same horizon and destination rules as the prediction. For destination \(k\), let \(Y_{ik}(h)=1\) if the individual transitions to \(k\) within \(h\), and let \(Y_{ik}(h)=0\) otherwise. A transition to another destination before \(h\) is a competing event and contributes zero to this destination-specific indicator when it rules out transition to \(k\) within the window. Remaining in the intermediate state through \(h\) also contributes zero.

For a prediction of the full distribution over possible states at the horizon, the outcome must instead identify the state occupied at \(T_i+h\), with competing destinations represented as distinct outcomes. The observed category and predicted probabilities must use the same categories. A transition probability to \(k\) and a probability of occupying state \(k\) at the horizon are different targets if the process permits leaving \(k\) before the horizon.

When all eligible entrants have fully observed outcomes through \(h\), a transition-specific Brier score is the mean squared difference between predicted probabilities and binary outcomes:

Here \(n\) is the number of eligible validation episodes with known outcomes at the horizon. A lower score means smaller average squared prediction error for this outcome in this validation sample; it does not by itself show why predictions perform as they do.

Censoring After Entry

A person lost to follow-up before \(h\) has an unknown outcome unless the event status is otherwise ascertained. Treating that person as event-free is incorrect. One option is to restrict analysis to entrants whose outcomes are fully known, but this can select a biased subset if completeness of follow-up is related to prognosis. Another option is inverse-probability-of-censoring weighting (IPCW), using the probability of remaining uncensored to the relevant event time or horizon.

For a simple IPCW Brier calculation, let \(G_i(u)\) be the probability that entrant \(i\) remains uncensored through \(u\), conditional on the observed information used in the censoring model. An observed transition to \(k\) at time \(u\le h\) receives squared-error contribution \((1-p_{ik}(h))^2/G_i(u-)\). An observed competing transition by \(h\) receives \(p_{ik}(h)^2/G_i(u-)\). Someone known to be event-free at \(h\) receives \(p_{ik}(h)^2/G_i(h)\). A person censored before \(h\) without an earlier event has no directly observed outcome contribution. Averaging these weighted contributions over the original eligible sample estimates the target score under the assumptions below.

This approach requires censoring to be conditionally independent of the relevant future outcome given the information in the censoring model, and positive probability of remaining uncensored over the required times in each relevant covariate group. The censoring model should be fitted and checked in the post-entry validation population. Large or highly variable weights indicate weak support and can make the estimate unstable. Censoring adjustment cannot repair an incorrect entry definition, an unavailable prediction, or an outcome window that was assigned inconsistently.

Why Conditioning on Entry Matters

Let \(A_i=1\) indicate that individual \(i\) enters the specified state and meets the entry-time eligibility rules. The prediction target is a conditional quantity, such as \(P\{Y_{ik}(h)=1\mid A_i=1,\text{information at }T_i\}\). Validation should therefore compare predictions with outcomes in the population represented by \(A_i=1\), not in an initial cohort that includes \(A_i=0\).

The reason follows directly from averaging. A mean prediction error among all baseline subjects combines errors for people who can receive the entry-time prediction with outcomes for people who cannot. Even if one were to assign a value to their unavailable predictions, the resulting average would target a different mixture. Restricting to eligible entrants instead estimates the average error conditional on the prediction task being available. This conditioning also changes the case mix: entry may be associated with disease severity or earlier transitions, so baseline-cohort performance cannot simply be carried over to entrants.

An appropriate validation report states the number of eligible entries, the entry period and rules, the number with each observed destination, the number censored before the horizon, and the method used for incomplete outcomes. These counts help readers see whether the stated target is supported by the data.

Diagnostics, Alternatives, and Limitations

Before interpreting a metric, check that every included episode has a prediction timestamp, that all predictors were available by that timestamp, and that outcome follow-up begins at the same operational point for each episode. Compare entry dates and observed follow-up across included and censored episodes. Inspect whether any subgroup has little or no chance of being followed through the horizon; this is a positivity warning for IPCW. Confirm that competing transitions are coded as competing outcomes rather than silently treated as independent censoring.

The main alternatives answer different questions. A baseline-cohort validation evaluates predictions made at baseline for the baseline cohort; it is not a substitute for evaluating a prediction issued at intermediate-state entry. Complete-case evaluation is straightforward when nearly all entrants have complete follow-up and completeness is not selectively related to outcomes, but it may be biased otherwise. IPCW retains partially followed entrants under its assumptions, but depends on a credible censoring model and can be unstable with extreme weights. A full state-occupation score may be preferable when the target is the state occupied at the horizon rather than a single transition.

A state-entry validation set also has limits. Its results apply to the specified eligible entrants and entry period, not automatically to all patients, other entry definitions, or future calendar periods. Small numbers of entrants or rare destinations can make performance summaries noisy. If repeated entries are included, the target is entry episodes rather than unique people, and dependence must be respected when uncertainty is quantified. The metric summarizes predictive performance; it does not establish that a model-guided intervention would improve outcomes.