Internal-External Validation Across Centers

Why Rotate Centers?

A joint model may be developed using longitudinal measurements and event outcomes collected at several centers. Its predictions can appear accurate overall while performing quite differently from one center to another. Differences in case mix, measurement schedules, follow-up, or clinical practice can all matter. An internal-external validation analysis makes this variation visible by treating each center in turn as a held-out validation sample.

The word internal indicates that validation uses centers from the available multicenter data set; external indicates that each held-out center is kept out of the model-fitting process for its fold. This is not the same as randomly splitting patients across training and validation sets. A patient-level split can place patients from the same center on both sides, allowing center-specific features to carry over and obscuring how well predictions work in a center not used for fitting.

The purpose is not merely to produce one performance number. The center-specific results show both the typical performance across the observed centers and the degree to which it varies. They do not, on their own, establish how well the model will perform in every future center or population.

Definition and Validation Target

Suppose a prediction is made at landmark time \(s\) for a patient who is event-free at \(s\), using the patient’s longitudinal history available by \(s\). For a prespecified horizon \(\tau\), the target might be the probability of an event by \(s+\tau\), conditional on being event-free at \(s\). Write the prediction for patient \(i\) in held-out center \(k\) as \(\hat p_{ik}^{(-k)}(\tau\mid s)\). The superscript \((-k)\) indicates that center \(k\) was excluded when fitting the model for that fold.

The endpoint, landmark, horizon, prediction time, and performance measures should be specified before examining center-specific results. Otherwise, selecting a favorable horizon or metric after seeing the results can make performance look better than it is. The joint model must also be refitted within each fold: this includes estimating its longitudinal and event-model parameters, as well as any data-driven preprocessing, tuning, or predictor selection.

Assumptions and a Fold-by-Fold Procedure

The validation procedure depends on having comparable outcome definitions and follow-up rules across centers. It also assumes that the held-out center’s data are not used, directly or indirectly, to develop its fold-specific predictions. Patients within a center may have related outcomes; the center is the unit being held out, not a claim that patient observations are independent across all centers. A center must contain enough eligible patients and observed outcomes to estimate the chosen metrics with useful precision.

For horizon-specific evaluation, censoring before \(s+\tau\) can leave a patient’s event status at the horizon unknown. A censoring-adjusted metric, such as an inverse-probability-weighted metric, requires an appropriate censoring model and a condition such as independent censoring conditional on the variables used in that model. If that condition is implausible, or if estimated censoring probabilities are near zero, the metric may be biased or unstable. The censoring adjustment should be applied consistently across folds.

What to Measure in Each Held-Out Center

For discrimination at horizon \(\tau\), one option is a time-dependent area under the receiver operating characteristic curve. In a cumulative/dynamic definition, cases experience the event by \(s+\tau\), while controls remain event-free beyond that horizon. The AUC is the probability that a randomly selected case receives a higher predicted risk than a randomly selected control, with ties handled according to the chosen convention. It is a ranking measure: it does not say whether the numerical risks are accurate.

For calibration, an observed-to-expected event ratio compares the observed number of events by the horizon with the sum of predicted risks for the corresponding patients, with appropriate censoring adjustment when outcomes are not fully observed. A ratio near 1 indicates agreement in aggregate; a ratio above 1 indicates more observed events than expected, and a ratio below 1 indicates fewer. This summary does not show whether predictions are accurate across the full range of risk. A calibration slope, assessed using a suitable horizon-specific calibration analysis, describes whether predicted risks are too extreme or too compressed; its ideal value is 1.

These measures answer different questions. A center can have strong ranking but poor absolute risk calibration, or reasonable calibration on average but weak ranking. Report them separately and interpret them in the clinical units and at the horizon used for prediction.

Summarizing Between-Center Variability

Let \(\hat\theta_k\) be a validation metric for held-out center \(k\). A simple descriptive summary is the unweighted mean across centers and the sample standard deviation across centers:

The mean gives equal weight to each center, while the standard deviation describes the observed spread of center-specific estimates around that mean. The range is another useful descriptive summary. These summaries are not interchangeable with a pooled patient-level metric: a pooled metric gives more influence to centers with more eligible patients and can conceal poor performance in smaller centers.

There is also an important uncertainty distinction. Each \(\hat\theta_k\) has sampling uncertainty because it is estimated from a finite held-out sample. The observed spread across centers reflects both genuine differences in performance and this estimation noise. With few centers, the sample standard deviation is itself imprecise and should not be treated as a stable estimate of a universal distribution of center performance. Report the center-specific sample sizes, event counts, uncertainty intervals where appropriate, and the number of centers alongside the spread summary.

Diagnostics, Alternatives, and Limitations

Before interpreting the results, check that every center was genuinely held out in its fold and that the model converged. Inspect the number of eligible patients, events, censoring before the horizon, and the distribution of predicted risks in each center. Extremely sparse outcomes or nearly identical predictions can make metrics unstable or undefined. Check whether censoring weights are extreme and whether the censoring model is plausible. Large differences in follow-up or measurement schedules can also affect performance, even when the model-fitting code is identical.

Internal-external validation complements, rather than replaces, validation in an independent cohort. It uses available centers efficiently and reveals variation across them, but centers used as validation folds may still resemble the centers used to fit the other folds. A random patient-level cross-validation analysis instead estimates performance when patients are held out while their centers remain represented in training; that answers a different question and may understate the challenge of a new center. Neither approach makes an unsuitable outcome definition, poor data quality, or unrealistic censoring assumption harmless.

A further limitation is that observed center variation is not a direct explanation of why performance differs. Center-specific estimates can suggest where investigation is needed, but they cannot by themselves distinguish differences in case mix from differences in measurement practice, event ascertainment, or model fit. Keep the descriptive validation result separate from causal claims about center characteristics. The introductory material on external validation in an independent cohort provides the related setting in which a separate cohort, rather than centers rotated from the development data, is used for evaluation.

Common Pitfall

Another frequent mistake is to report only a pooled performance number. A pooled AUC or observed-to-expected ratio may look satisfactory while concealing one or more centers with notably poorer performance. Present the center-specific estimates and their uncertainty, then add a pooled or equal-center summary only after defining what that summary weights.