Diagnosing Censoring Weights In Transition Validation

Why Censoring Weights Need Their Own Diagnostic

An IPCW Brier score corrects for outcomes that are unknown because follow-up ended before the validation horizon. The correction relies on a model for remaining uncensored. If that model assigns very small probabilities to remaining observed, the corresponding inverse-probability weights can be large. A few such weights may dominate the score, making the result noisy or highly sensitive to modeling choices.

The diagnostic question is not simply whether weights are “large.” It is whether the censoring model is credible for the subjects and times that contribute to the metric, whether the data provide support across relevant covariate patterns, and whether a small number of observations controls the result. These checks concern the reliability of the weighted validation metric; they do not establish that the transition-prediction model itself is well calibrated or otherwise adequate.

What the Censoring Weights Represent

For a transition-specific Brier score at horizon \(t\), the outcome is whether the target transition has occurred by \(t\), under the transition definition being validated. If the target transition has occurred, that positive outcome is known. If a competing transition makes the target transition impossible by \(t\), the negative outcome is known at that competing transition. A subject with no determining transition who is observed through \(t\) has a known negative outcome at \(t\). A subject censored before the outcome is determined contributes no observed outcome at that horizon.

Write \(p_i(t)\) for the predicted probability of the target transition by \(t\), and \(Y_i(t)\) for its binary outcome when fully observed. Let \(A_i\) be the time at which the outcome is ascertained: the target or precluding competing transition time when one determines the outcome, or \(t\) when follow-up establishes the outcome through the horizon. Let \(R_i=1\) if the outcome is observed by that time and \(R_i=0\) otherwise. With the cohort size \(n\) as denominator, the IPCW Brier score is

Thus, a known negative determined by a competing transition is weighted according to the probability of remaining uncensored until that transition time, not automatically until \(t\). The outcome definition matters: if a competing transition does not rule out the target transition under the stated target, it does not by itself ascertain a negative outcome. The analysis must use the time at which the specified outcome actually becomes known.

Assumptions Behind the Diagnostics

Weight diagnostics are meaningful only in relation to the assumptions that justify the weighting. The key condition is conditional independent censoring: given \(H_i\), censoring must not carry additional information about the transition outcome process being validated. The conditioning history should include relevant predictors of both censoring and the transition process, measured at the appropriate prediction origin. If censoring depends on an unrecorded marker of impending transition, weighting on recorded history alone may not remove the resulting bias.

A second requirement is positivity: within covariate and history patterns represented in the target validation population, the probability of remaining uncensored until the outcome-ascertainment time must be positive. In practice, probabilities near zero are also problematic. Even if they are not exactly zero, their reciprocals can make the estimate depend heavily on a few records. The censoring model must also be adequately specified and estimated; a smooth or plausible-looking weight distribution does not prove that its conditional probabilities are correct.

Censoring-model choices should match the data structure. For example, the model may need to account for time, covariates, and changes in observed multi-state history that predict censoring. Assess whether predicted censoring survival probabilities are calibrated in relevant time intervals and covariate or history groups, and whether observed censoring patterns are reproduced. A model fitted only to overall censoring frequency can miss important conditional differences.

A Short Justification for Inverse-Probability Weighting

The weighting argument explains why a weight distribution with extreme values is a substantive concern, not just an inconvenient summary. Suppose the outcome is ascertained at \(A_i\), and conditional on the relevant complete transition history and \(H_i\), censoring is independent of that history. Then the probability of observing the outcome at \(A_i\) is \(G(A_i\mid H_i)\). Conditional on the complete history,

The equality uses \(E(R_i\mid \text{complete history},H_i)=G(A_i\mid H_i)\). Averaging over subjects therefore recovers the complete-outcome mean loss in expectation, under the stated assumptions and a correctly specified censoring survival function. If \(G\) is close to zero, the factor \(1/G\) is large: a small number of observed subjects must represent many similar subjects whose outcomes were not observed. If \(G=0\) in a target subgroup, no inverse weight can recover that subgroup’s outcome distribution from the data.

Inspecting Weight Size, Concentration, and Support

At minimum, examine the distribution of \(\widehat{G}(A_i\mid H_i)\) and its inverse among records whose outcomes are observed. Report useful quantiles, the maximum, and the number of records with probabilities below prespecified thresholds. Inspect these summaries by ascertainment time and by important covariate or state-history groups. An acceptable overall average censoring probability can conceal a subgroup with almost no chance of observed follow-up.

The effective sample size is a descriptive measure of weight concentration. For observed records with weights \(w_i=1/\widehat{G}(A_i\mid H_i)\), one common form is

It equals the number of records when all weights are equal and decreases as weights become more concentrated. It is not a replacement sample size for inference, nor a universal pass/fail threshold. Also inspect each weighted loss \(w_i\{Y_i(t)-p_i(t)\}^2\): a subject with a large weight may have a small loss, while a moderately large weight paired with a large loss may dominate the score.

Positivity and Truncation Sensitivity

Positivity should be assessed for the actual times at which outcomes are ascertained, not only at the nominal horizon. Plot or tabulate \(\widehat{G}(u\mid H)\) across time and relevant histories. Look for probabilities that approach zero, groups with no observed follow-up to the needed time, and sharp changes caused by sparse data. When support is absent, the metric for the full target population is not identified by ordinary IPCW from those data. Possible responses include narrowing the target population or horizon to regions with support, improving data collection, or using a defensible censoring model with adequate information. Extrapolating a fitted model into an unsupported region does not create empirical positivity.

Truncation caps large weights, for example replacing \(w_i\) with \(\min(w_i,c)\) for a chosen cap \(c\). This often reduces variance, but it also reduces the upweighting of subjects with low estimated observation probabilities. The resulting estimator is generally no longer the same unbiased IPCW estimator for the original target; it trades some protection against instability for potential truncation bias. Report the untruncated estimate, the cap or rule, the fraction of weights affected, and the sensitivity of the metric to reasonable alternatives. Truncation is not a cure for a misspecified censoring model or structural lack of positivity.

Diagnostics, Alternatives, and Common Pitfalls

Assess censoring-model fit by comparing predicted and observed censoring patterns over time and across groups relevant to the validation target. Examine whether key predictors and history terms are included, whether the fitted probabilities are plausible, and whether conclusions change under reasonable model specifications. Sparse groups, separation, unstable time effects, or marked mismatch between predicted and observed censoring are warnings. In a small validation sample, these checks may have limited power; failure to detect a mismatch is not proof of correct specification.

Complete-case validation, which scores only subjects observed through the horizon without weighting, is simpler but can be biased when the probability of remaining observed is related to outcomes or predictions. IPCW instead reweights observed outcomes under conditional independent censoring and positivity, but it can be variable and depends on the censoring model. A restricted-horizon analysis can avoid regions with near-zero observation probabilities, but it answers a narrower question. Truncated IPCW can stabilize a calculation while changing its finite-sample behavior and potentially its target. These approaches address different compromises; none makes unsupported data informative without additional assumptions.

• Pitfall: interpreting a large weight as proof of bias. A large weight indicates a small estimated observation probability and potential instability; by itself, it does not establish bias. Check model fit, support, and the weighted loss.
• Pitfall: treating weight truncation as a harmless numerical fix. Capping changes the estimating procedure and can introduce bias. Report it as a sensitivity analysis, not as if it were the unmodified IPCW score.
• Pitfall: judging positivity from the average weight. A moderate overall average can hide unsupported histories. Examine probabilities and weights within relevant time and history groups.
• Pitfall: confusing censoring diagnostics with prediction diagnostics. Acceptable censoring weights do not demonstrate good transition predictions; poor weights do not by themselves show that predicted probabilities are poor.

The introductory material on IPCW Brier scores under right censoring gives the basic weighted-score setup. Here the emphasis is on whether the censoring weights make a transition-validation result credible: their assumptions, concentration, support, and sensitivity are part of the metric’s interpretation.