Designing Validation Cohorts For Adequate Transition Events

Why Plan Around Transitions?

A validation cohort can be large yet provide little information about a particular transition. If few people enter the relevant origin state, the transition is rare, or follow-up is often censored before the prediction horizon, a transition-specific calibration estimate may be imprecise or unstable. Planning should therefore start with the transitions and time horizons that matter, then translate precision goals into expected observed information and a cohort size.

The target is not simply a total number of participants. It is a sufficient number of subjects whose transition outcomes can be estimated for the validation quantity of interest. The earlier material on rare-transition validation and transition-specific confidence intervals motivates this planning focus. There is no universal number of events that makes every calibration analysis adequate: required information depends on the estimand, prediction distribution, censoring, and desired precision.

Translate a Precision Goal Into an Evaluability Target

For a simple starting point, suppose every subject has a fully observed binary outcome \(Y_i\), where \(Y_i=1\) means the target transition occurred by \(\tau\). The event probability is \(q=P(Y_i=1)\), and its sample proportion has approximate standard error \(\sqrt{q(1-q)/m}\) when \(m\) independent outcomes are fully observed. A normal-approximation confidence interval with desired half-width \(d\) therefore suggests the evaluable sample size

Here \(z_{1-\alpha/2}\) is the standard normal quantile for the chosen confidence level. This calculation is a planning approximation, not a guarantee of realized interval width. If \(q\) is uncertain, a conservative choice for a binary outcome is \(q=0.5\), which maximizes \(q(1-q)\). A validation target may instead specify precision for a calibration difference, slope, or another metric; its variance need not equal the variance of a single event proportion. Use a variance formula appropriate to that estimand when available, and treat the simple proportion calculation as a transparent benchmark.

When outcomes are censored, \(m\) is not automatically the cohort size. A subject censored before \(\tau\) may contribute partial information under a suitable estimator, but cannot simply be counted as a known non-event. The sample-size calculation must reflect how the planned analysis handles censoring, including any inverse-probability weighting or pseudo-observation approach.

Account for Censoring and the Predictor Distribution

Let \(X\) denote baseline predictors used in the validation analysis. For a target transition from an origin state, let \(F_j(t\mid X)\) denote the conditional cumulative incidence of that transition by time \(t\), with competing destinations treated as distinct outcomes. Let \(C\) be the censoring time and \(G(t\mid X)=P(C\geq t\mid X)\) the conditional probability of remaining uncensored through \(t\). Under conditional independence of censoring and the event process given \(X\), the probability of an observed target transition by \(\tau\), conditional on \(X\), is

The overall probability is the average over the planned validation population's predictor distribution:

The left limit \(G(u-\mid X)\) reflects being uncensored immediately before time \(u\). These expressions require the stated conditional independence assumption and a clear censoring model. In particular, the marginal formula \(\int_0^\tau G(u-)\,dF_j(u)\) is justified if censoring and the event process are independent marginally; it does not follow merely from independence conditional on predictors. When censoring is only conditionally independent, average the conditional products as shown, or specify the planned adjustment. For transitions that can occur only after entering an intermediate state, the corresponding calculation must also account for reaching that state and remaining under observation while at risk for the transition.

For a cohort of size \(n\), the expected number of observed target transitions is \(n\pi_j^{\mathrm{obs}}(\tau)\), if participants are sampled independently from the planned population and the probabilities are correctly specified. Dividing a desired expected event count by \(\pi_j^{\mathrm{obs}}(\tau)\) gives an initial sample-size estimate. This is an expectation, not a promise that the realized count will meet the target.

Design for the Transitions That Matter

In a multi-state validation, several transitions may be evaluated, and their event probabilities can differ substantially. For each transition, horizon, and important subgroup, calculate an expected observed event probability under the planned case mix and censoring. Then compare the sample sizes implied by event-count and precision targets. If several targets must be met, the most demanding one often determines the minimum cohort size, but subgroup allocations may require deliberate oversampling or stratified recruitment.

Practical Checks, Alternatives, and Limits

A planning calculation is only as credible as its inputs. Use external data or a pilot cohort to estimate transition probabilities, predictor distributions, and censoring patterns. Where possible, calculate conditional quantities by predictors that affect both event risk and censoring, then average them over the intended validation population. If the validation case mix differs from the data used to estimate those quantities, use the planned case mix rather than carrying over an unadjusted marginal rate.

Check at least the following before fixing the cohort size:

• Horizon and transition definition: confirm the origin, destination, and time horizon. A different horizon can change both event frequency and censoring burden.
• Competing destinations: classify observed competing transitions as competing outcomes, not as censored target events.
• Censoring assumptions: verify that the planned conditional censoring model is plausible and that the predictors needed for adjustment will be available.
• Estimand-specific variance: do not substitute a binary event-proportion formula for a calibration slope or another metric without justification.
• Realized counts: account for random variation around expected event counts. If falling below a minimum count would be costly, plan above the expected-count threshold and quantify the chance of shortfall.
• Subgroups and clustering: allow for smaller effective information when estimates are needed within groups or when observations are dependent.

A direct event-count calculation is easy to explain, but it cannot by itself establish a desired confidence-interval width for every calibration metric. A precision-based calculation is better aligned with an explicit estimand but may need a variance formula that incorporates weighting, prediction variation, and nuisance-parameter estimation. When those components are complex or poorly approximated, a design-specific simulation can evaluate interval width and failure rates under plausible data-generating scenarios. That approach complements analytic calculations; it does not eliminate the need to state and check assumptions.

Diagnostics during validation should compare planned with observed transition counts, follow-up, censoring, and predictor composition. A shortfall can arise because the event rate was lower than expected, censoring was heavier or differently patterned, or recruitment produced a different case mix. If inverse-probability weights are used, inspect their distribution and the effective sample size: a nominally large cohort can carry much less information when a few subjects receive extreme weights. Sparse transitions, unstable estimates, very wide intervals, or strong sensitivity to a small number of subjects are signs that the planned information was not achieved. These checks help explain limitations; they do not justify treating unknown outcomes as known non-events.

Common Mistake

Wrong reasoning: “The overall target-transition probability is 20%, so 250 participants will yield 50 observed transitions.” This calculation ignores censoring and assumes every target event is observed. A related error is multiplying a marginal event probability by a marginal uncensored probability when censoring is independent only conditional on predictors.

Correction: Calculate the conditional probability of an observed target transition, combine conditional event and censoring distributions at each predictor value, and average over the intended validation population. Then use \(n\pi_j^{\mathrm{obs}}(\tau)\) for the expected observed count, while recognizing that actual counts vary. Verify that the precision calculation corresponds to the calibration estimand and planned estimator.