Fair Head-to-Head Evaluation of Landmark and Joint Models

Why the Comparison Must Be Matched

A head-to-head comparison can appear to show that one prediction method performs better when the actual difference comes from who was evaluated, when predictions were made, or how outcomes were scored. Landmark and joint models can use longitudinal information differently, but a fair comparison asks both to predict the same outcome for the same patients at the same prediction time and horizon. Their predictions must then be assessed with the same validation procedure.

For a prediction time \(s\) and horizon \(h\), the target here is survival through \(s+h\), conditional on being alive and event-free at \(s\). Let \(T_i\) be patient \(i\)'s event time, and let \(\mathcal H_i(s)\) denote that patient's longitudinal history available by \(s\). Both methods produce a predicted probability for the same patient-specific conditional event: survival beyond \(s+h\), given survival through \(s\) and information available at \(s\). A model's prediction must not use measurements recorded after \(s\).

This target is conditional on landmark survival. It is not a comparison of unconditional baseline predictions in the full cohort. The earlier material on landmarking and joint models distinguishes their conditioning sets and predictor information; for evaluation, predictions must nevertheless be placed on a common conditional target. In particular, a joint model's prediction should be calculated conditional on the patient being event-free at \(s\) and using only the history available by \(s\), not treated as a baseline prediction for a different population.

Set the Evaluation Target Before Comparing Predictions

For each selected landmark \(s\), define the eligible set as the validation patients who are alive and event-free at \(s\), with the required covariate and longitudinal information available under the prespecified rules. At horizon \(h\), the outcome is whether the event occurs by \(s+h\). Define \(Y_i(s,h)=1\) if patient \(i\) remains event-free beyond \(s+h\), and \(Y_i(s,h)=0\) if the event occurs between \(s\) and \(s+h\). For a patient censored before the horizon without an event, this outcome is unknown, not zero.

Write \(\hat S_i^{LM}(s+h\mid s)\) and \(\hat S_i^{JM}(s+h\mid s)\) for the landmark and joint-model predicted survival probabilities, respectively. The conditioning notation emphasizes the common landmark target: probability of surviving past \(s+h\), conditional on survival to \(s\) and the information each model is allowed to use by \(s\). If the comparison concerns event risk rather than survival probability, use \(1-\hat S_i(s+h\mid s)\) for both methods and define the outcome consistently.

Before looking at validation performance, prespecify the following items:

• Eligible patients: use the same landmark risk set and the same rules for missing or unavailable measurements.
• Prediction times and horizons: compare at identical \(s\) and \(h\), with the same time origin and event-free requirement at \(s\).
• Outcome: apply the same event definition and handling of competing events, if relevant. Do not silently treat a competing event as ordinary censoring unless the estimand and method justify that choice.
• Information cutoff: restrict both models to predictors observed by the prediction time. Any differences in how models summarize permitted history are part of the model comparison, not a reason to give one method future information.
• Validation and scoring: use the same held-out patients, scoring rule, and censoring adjustment for both sets of predictions.

Scoring With Right-Censored Outcomes

When every outcome is observed, the Brier score is the average squared difference between a predicted probability and its binary outcome. With right censoring before the horizon, some outcomes are unknown, so simply dropping censored patients or coding them as event-free can distort the score. An inverse-probability-of-censoring-weighted (IPCW) Brier score corrects for the fact that outcomes are observed with different probabilities.

The event weight uses the censoring survival probability just before the event time. A patient with an event observed by the horizon has a known outcome even though follow-up ends at the event. A patient known event-free through the horizon also has a known outcome. A patient censored earlier without an event contributes zero because the outcome is unavailable, but remains included in the denominator \(N\). Under the assumptions below, weighting the observed outcomes corrects for their differing probabilities of observation.

For this score to have its intended interpretation, assume censoring is independent of the event time conditional on the variables used in the censoring model, within the landmark risk set; the censoring model estimates \(G_i(u)\) adequately; and \(G_i(u)>0\) over the event times and horizon being evaluated. The last condition is a positivity requirement: if some patients have essentially no chance of remaining observed, their outcomes cannot be reliably recovered by weighting. These assumptions concern the censoring adjustment, not whether either prediction model is correctly specified.

Estimate or specify the censoring distribution using the same procedure for both models. It may be estimated in the validation data using follow-up and permitted covariates, or by a prespecified approach using training data; the choice should be fixed before comparison. Very small estimates of \(G_i\) create large weights and unstable scores. Report weight diagnostics, and consider a prespecified truncation or restriction when extreme weights are a concern, noting that this changes the evaluation procedure.

A Paired Calculation of Prediction Performance

Why the Weighting Works, and How to Quantify the Difference

The IPCW calculation is motivated by the probability that an outcome is observed. For a patient who remains event-free to the horizon, the outcome is observed only if censoring follow-up lasts at least \(h\). Let \(L=(1-\hat S)^2\) denote that patient's squared prediction error when \(Y=1\). Under conditional independent censoring, the observation indicator has conditional probability \(G_i(h)\). Therefore, the conditional expected weighted contribution is

For an event observed at lag \(u\leq h\), the event outcome is observed if censoring has not occurred before \(u\). Its observation probability is \(G_i(u-)\); dividing its squared error by that probability gives the same correction. This argument explains the role of the weights, but does not make the score reliable when censoring is informative after conditioning, the censoring model is poor, or the weights are extreme.

Since both models are scored on the same patients with the same weights, the natural contrast is paired: calculate each patient's weighted loss under each model, subtract the landmark loss from the joint-model loss, and summarize those differences. A lower average Brier score indicates better overall probabilistic accuracy for that target and horizon; it combines aspects of calibration and discrimination. It is not itself a measure of discrimination alone. The earlier material on discrimination of landmark predictions by horizon treats that distinct performance dimension.

Uncertainty should respect the pairing. A bootstrap can resample patients, carrying both predictions, outcomes, and weights together, then recompute the score difference. If the evaluation includes model fitting, resampling or repeated validation should also reflect training-sample variability; bootstrapping only a fixed validation set quantifies uncertainty conditional on the fitted predictions and the specified censoring adjustment. With repeated prediction times per patient, resample patients as clusters, not individual patient-time records, so within-patient dependence is retained.

Multiple Horizons, Diagnostics, and Alternatives

Useful checks include verifying that the two prediction columns have the same patient-time rows, that predictions are valid probabilities, and that no predictor measured after \(s\) entered either prediction. Tabulate eligibility, events, and censoring by horizon; inspect the estimated \(G_i\) values and the largest IPCW weights; and examine whether a small number of patients dominate the score. For each horizon, confirm that the outcome coding agrees with the probability being scored: survival probability must be paired with \(Y=1\) for survival, not with an event indicator.

A Brier score is one suitable common score, but not the only alternative. Calibration summaries assess agreement between predicted and observed risks; discrimination measures how well predictions rank patients with different outcomes. These quantities answer different questions, and none should be substituted for another without explanation. If a different score is used, apply the same score and censoring adjustment to both methods and state the target estimand. A comparison of each method using its own preferred metric, cohort, or censoring procedure is not a head-to-head comparison.

The main limitations remain important even with careful matching. IPCW depends on assumptions about censoring and can be unstable under weak follow-up. A score at one horizon does not establish performance at another. A single validation sample can yield a noisy difference, especially with few events or few eligible patients. Finally, fair evaluation does not ensure that either model is well calibrated, correctly specified, or transportable to a different population. The comparison is only as useful as its target, validation design, and uncertainty assessment.

Common Pitfall

A second frequent error is to remove early-censored patients from the denominator or code them as event-free. Neither is appropriate for the IPCW score defined here: their unknown outcome receives zero weighted contribution, while the denominator remains the full eligible \(N\). The censoring adjustment, rather than an invented outcome, accounts for their incomplete follow-up.