Why a Worked Validation Analysis Needs a Fixed Target
A dynamic joint model uses longitudinal history available up to a prediction time to estimate a patient’s future event risk. Validating that prediction requires more than comparing a model score with eventual outcomes: the cohort, prediction time, horizon, and handling of incomplete follow-up must all match the intended use. Otherwise, an apparently strong performance estimate may answer a different question from the one clinicians would ask.
This tutorial follows one hypothetical analysis. At a landmark time \(s\), the target is the probability of the event during the next \(w\) time units, conditional on being event-free at \(s\) and on longitudinal and clinical information available by \(s\). The event horizon is \(s+w\). The model and its prediction procedure are frozen before evaluation in the validation cohort.
This is a landmark estimand: it concerns people known to be event-free and under observation at \(s\), not everyone originally enrolled. The model must use only information available by that time. The prediction horizon must also be chosen in advance; changing it after seeing validation outcomes changes the performance question.
Define the Validation Cohort Before Looking at Performance
Begin with a protocol that names the target population, landmark, horizon, outcome, and prediction method. Include a person only if event-free status at \(s\) can be established and the information needed to generate a prediction is available under the prespecified missing-data procedure. Measurements collected after \(s\), and summaries calculated using future follow-up, must not enter the predictor history.
For example, suppose a hypothetical study enrolled 1,000 people. By month 12, 120 had experienced the event and 40 had been censored before month 12. The landmark cohort therefore contains \(1{,}000-120-40=840\) people known to be event-free and still under observation at month 12. If a prespecified rule cannot generate predictions for 12 of them because required history is unavailable, the analysis includes 828 people and explicitly describes this evaluable target population. It should not silently count the 12 as low-risk or event-free.
Worked Example: Applying the Landmark Cohort Rules
Consider the hypothetical month-12 analysis above, with a prediction horizon ending at month 36. A participant who experienced the event at month 9 is excluded because the person was not event-free at the landmark. A participant censored at month 10 is excluded because month-12 eligibility cannot be established. A participant event-free and under observation at month 12 is eligible if the model can produce a prediction from information available by month 12.
Suppose participant R has marker measurements at months 0, 6, and 12, and no event by month 12. These measurements may be used if they were available at the time the prediction would have been issued. A measurement recorded at month 15 cannot be used, even if it is later used to improve a retrospective marker trajectory estimate. Thus R contributes one prediction for the interval \((12,36]\), not a prediction retrospectively informed by the future.
The arithmetic is \(1{,}000-120-40=840\) landmark-eligible participants, followed by \(840-12=828\) evaluable predictions under the specified history rule. The analysis population is therefore 828 people, and its scope is the eligible people for whom the stated prediction procedure can operate.
Adjust for Censoring at the Prediction Horizon
At the end of follow-up, some eligible people will have an observed event before month 36, some will be observed event-free through month 36, and some will be censored earlier without a known horizon outcome. The third group cannot simply be labelled event-free. A standard approach is inverse-probability-of-censoring weighting (IPCW): participants whose outcome is observed receive weights that compensate for similar participants whose outcomes were censored.
Let \(G_i(t)=P(C_i-s\geq t\mid C_i>s,Z_i)\) denote the probability of remaining uncensored for at least \(t\) time units after the landmark, conditional on suitable observed predictors \(Z_i\). For an event at elapsed time \(t_i=T_i-s\), the event weight is \(1/G_i(t_i-)\); for a participant known event-free through \(w\), the control weight is \(1/G_i(w)\). The left limit at the event time accounts for the convention used to handle censoring at an event time. Participants censored before \(w\) with no event are not classified as controls.
A short justification explains the weighting. If an outcome contribution \(f_i\) is observed only when the participant remains uncensored through its ascertainment time \(t\), let \(R_i(t)\) indicate that it is observed. Under the stated conditional independence assumption, the probability of \(R_i(t)=1\), given the relevant observed information, is \(G_i(t)\). Consequently,
Thus the weighted contribution has the same conditional expectation as the contribution that would have been available with complete outcome follow-up. This argument depends on the censoring assumptions; weighting does not repair informative censoring left unexplained by the variables in the censoring model.
Worked Example: Censoring-Adjusted Horizon Calibration
In a small hypothetical validation sample at month 12, six participants have known outcomes by month 36. Three experienced the event before the horizon, with censoring survival probabilities \(0.90\), \(0.80\), and \(0.90\). Three were observed event-free through the horizon, each with censoring survival probability \(0.80\). A seventh participant was censored before month 36 without an event and is not assigned a horizon outcome.
The event weights are \(1/0.90=1.111\), \(1/0.80=1.250\), and \(1/0.90=1.111\). Each known event-free participant receives \(1/0.80=1.250\). The sum of the event weights is \(1.111+1.250+1.111=3.472\); the sum of all six observed-outcome weights is \(3.472+3(1.250)=7.222\).
The IPCW estimate of the event risk is therefore \(3.472/7.222=0.4808\), or about 48.1%. This is an estimate of the event risk in the landmark target population under the censoring assumptions, not the raw fraction of observed outcomes that were events. The censored seventh participant contributes neither a known event nor a known event-free outcome.
Evaluate Horizon-Specific Discrimination
For a single event type, a cumulative/dynamic AUC at horizon \(w\) compares people who experience the event by \(s+w\) (cases) with people event-free beyond \(s+w\) (controls). It is the probability that a randomly selected case has a higher predicted risk than a randomly selected control, with half credit for tied predictions. It is specific to the landmark and horizon; it does not describe discrimination at every follow-up time.
When censoring is present, estimate this comparison using the IPCW weights for cases and controls. For case set \(\mathcal C\), control set \(\mathcal N\), and weights \(a_i\) defined above, the weighted AUC is
Here \(I(\cdot)\) is 1 when its condition is true and 0 otherwise. The formula gives more influence to observed cases and controls who represent participants with a lower probability of remaining uncensored. It does not turn early-censored participants into controls.
Worked Example: Computing the Weighted AUC
Use the six known outcomes from the preceding hypothetical example. The three cases have predicted risks \(0.60\), \(0.40\), and \(0.70\), with weights \(1.111\), \(1.250\), and \(1.111\), respectively. The three controls have predicted risks \(0.20\), \(0.30\), and \(0.50\), each with weight \(1.250\). There are no tied risks.
The total case weight is \(1.111+1.250+1.111=3.472\), and the total control weight is \(3(1.250)=3.750\). The denominator of the AUC is \(3.472(3.750)=13.020\), up to rounding. The case with risk \(0.60\) outranks all controls, contributing \(1.111(3.750)=4.167\). The case with risk \(0.40\) outranks the controls at \(0.20\) and \(0.30\), contributing \(1.250(2.500)=3.125\). The case with risk \(0.70\) outranks all controls, contributing \(1.111(3.750)=4.167\).
The numerator is \(4.167+3.125+4.167=11.459\), so the weighted AUC is \(11.459/13.020=0.880\), allowing for rounding. In this hypothetical sample, a randomly selected case-control pair is correctly ordered with estimated probability about 0.88, with ties counted as half if any occurred. This is not evidence that the model is well calibrated: discrimination and calibration are different properties.
Assess Calibration and Quantify Uncertainty
Calibration asks whether predicted risks agree with observed event frequencies. One useful summary is calibration-in-the-large: compare the average predicted risk with the censoring-adjusted observed risk in the same target population. For the six-person illustration, the predictions for cases are \(0.60,0.40,0.70\), and those for controls are \(0.20,0.30,0.50\). Applying the outcome-observation weights gives a weighted mean prediction of
The observed weighted risk was \(0.481\), so observed risk minus mean predicted risk is \(0.481-0.442=0.039\), or about 3.9 percentage points. In this small illustration, the predictions are lower on average than the adjusted observed risk. The difference alone cannot show whether the discrepancy is systematic, and it does not describe calibration across the risk range. A fuller assessment can compare adjusted observed and predicted risks in prespecified risk groups or use a calibration plot, while reporting the uncertainty and the number of participants represented in each part of the plot.
Estimate uncertainty by resampling participants, not individual measurement rows: repeated measurements from one person belong together. In each bootstrap replicate, resample people from the landmark cohort and recompute the censoring model, weights, and performance measures. If validation is clustered by center, preserve the relevant center structure or resample centers when the inferential target requires it. A percentile interval uses the selected lower and upper quantiles of the bootstrap estimates.
Worked Example: Interpreting Bootstrap Uncertainty
Suppose a hypothetical analysis of a much larger validation cohort uses 1,000 subject-level bootstrap replicates and obtains a 95% percentile interval of \(0.76\) to \(0.93\) for the horizon-specific AUC estimate. The interval describes uncertainty in the estimated AUC due to sampling the validation population, under the bootstrap design and the analysis assumptions. It does not mean there is a 95% probability that the fixed population AUC lies inside this particular interval.
If the fitted joint model was developed elsewhere and held fixed, this bootstrap interval primarily reflects validation-sample uncertainty, including uncertainty from estimating censoring weights if the censoring model is refitted in each replicate. It does not include uncertainty from developing the prediction model. To represent the full development-and-validation procedure, resampling must also repeat model fitting and every data-dependent step of that procedure; this generally requires an appropriate resampling design for the development data as well.
Diagnostics, Alternatives, and Common Failure Modes
Before interpreting a performance estimate, check whether the analysis behaves as its assumptions require. Report how many people enter the landmark cohort, how many have observed events, how many are known event-free through the horizon, and how many are censored earlier. Inspect the censoring survival estimates and the distribution of IPCW weights; a few extreme weights can dominate an estimate. Compare key predictors between people with complete horizon outcomes and those censored earlier, while remembering that no comparison can prove conditional independent censoring.
The main alternative is a complete-case analysis restricted to participants whose outcomes are known at the horizon. It is simple, but estimates performance in that observed subset; it may not represent the landmark cohort if follow-up completeness is related to risk. Kaplan–Meier-based observed-risk estimates are another standard approach when their censoring structure is appropriate, but they do not by themselves provide the case-control weighting formula shown here for discrimination. Choice of approach should match the estimand and censoring structure, rather than convenience.
- Wrong: Count everyone censored before the horizon as event-free. Correction: Their horizon outcome is unknown; exclude them from known case/control status and use an appropriate censoring adjustment.
- Wrong: Use marker values recorded after the landmark because they improve the prediction. Correction: A prediction at \(s\) can use only information available by \(s\).
- Wrong: Treat a high AUC as proof that predicted probabilities are accurate. Correction: AUC measures ranking; assess calibration separately.
- Wrong: Apply IPCW without examining the weights. Correction: Positivity violations or a misspecified censoring model can make weighted estimates unstable or biased.
No single summary establishes that a model is suitable for use. Estimates depend on the landmark population, prediction horizon, censoring assumptions, and whether the validation sample represents the intended setting. If censoring depends on unmeasured prognosis after conditioning on the variables in the censoring model, IPCW may not remove bias. A small number of events, substantial weight variability, or narrow risk ranges also limits what calibration and discrimination analyses can establish.
Check Your Understanding
Answer each question, then compare with the brief solutions.
- At a month-10 landmark, why is a person censored at month 8 not part of the landmark risk set?
- At a 12-month prediction horizon, what should be done with a person censored after 7 months without an event?
- In a cumulative/dynamic AUC, who are the cases and who are the controls?
- A model has a high horizon-specific AUC but its weighted observed risk exceeds its weighted mean predicted risk. What does that comparison indicate, and what does it not establish?
- What does a subject-level bootstrap interval for a frozen model’s AUC capture, and what uncertainty may it omit?
Solutions. (1) Event-free status at month 10 is unknown, so landmark eligibility cannot be established. (2) Do not label the person event-free through 12 months; treat the outcome as censored and use a suitable censoring adjustment under its assumptions. (3) Cases experience the event by the horizon; controls are known event-free beyond it. (4) It indicates underprediction on average in the evaluated population, but does not establish the pattern of calibration across risk values or imply poor discrimination. (5) It captures validation-sample uncertainty under the bootstrap design, including censoring-model uncertainty if refitted; it may omit uncertainty from developing and fitting the prediction model.