Why Censoring Changes the Brier Score
A Brier score compares predicted probabilities with observed outcomes at a specified horizon. Under right censoring, some subjects have not been followed long enough to determine their transition status at that horizon. Scoring only subjects whose status is known can distort the result: the observed subset may differ systematically from the full validation sample. Inverse-probability-of-censoring weighting (IPCW) corrects this selection by giving more weight to observed outcomes that had a lower chance of being observed.
Consider a prediction made at a fixed origin for horizon \(\tau\). Suppose there are \(K\) competing outgoing transition types of interest. The outcome is category \(k\) if transition type \(k\) occurs by \(\tau\), for \(k=1,\ldots,K\), and category 0 if none of these transitions occurs by \(\tau\). The prediction for subject \(i\) is the vector \((p_{i0},p_{i1},\ldots,p_{iK})\), with nonnegative entries summing to one. The complete-data Brier loss is the sum of squared differences between this vector and the one-hot outcome vector.
This definition treats category 0 as no relevant transition by the horizon. A transition of a competing type is therefore a positive outcome for its own category and a negative outcome for the other categories. Predictions must refer to the same origin, horizon, and outcome categories as the observed status. This tutorial concerns the fixed-horizon score, not how the transition probabilities are estimated.
IPCW Contributions and Assumptions
Write \(G_{>}(t\mid Z_i)=P(C_i>t\mid Z_i)\), where \(Z_i\) denotes observed covariates used to model censoring. The strict inequality is important: the observation rules below use strict survival beyond the event time or horizon. An event of type \(j\) at or before \(\tau\) is observed when \(\delta_i=1\); a no-transition outcome is counted as observed only when \(U_i>\tau\), which requires both \(T_i>\tau\) and \(C_i>\tau\). A subject censored before or exactly at \(\tau\), without an earlier observed transition, contributes no outcome loss under this convention.
The assumptions for the basic argument are: censoring is conditionally independent of \((T_i,J_i)\) given \(Z_i\); the censoring survival probabilities are correctly known or consistently estimated; and positivity holds, so \(G_{>}(t\mid Z_i)>0\) at event times and horizons that contribute. For the observed-transition indicator, assume no ties between transition and censoring times; equivalently, the event-observation rule and censoring survival convention must handle ties consistently. Predictions and the censoring model must be defined for the validation subjects. If predictions are fitted using the same data, the validation design must account for that fitting, for example through a separate test sample or suitable resampling.
With the true censoring distribution known, the subject-level IPCW contribution is
The bracketed quantities are the complete-data Brier losses for the corresponding observed outcome category. The empirical IPCW Brier score is \(\widehat{BS}_{IPCW}(\tau)=n^{-1}\sum_{i=1}^n W_i(\tau)\). A censored subject with unknown status contributes zero, not a zero-valued Brier loss: the inverse-probability weights on observed outcomes compensate for the unobserved outcomes in expectation.
Why the Weighting Works
The key justification is an inverse-probability identity. Conditional on a subject’s covariates and their full transition outcome, the event indicator is observed with probability \(G_{>}(T_i\mid Z_i)\), while a no-transition outcome is observed under the strict rule with probability \(G_{>}(\tau\mid Z_i)\). Conditional independence makes these probabilities the censoring probabilities alone.
Given \(T_i=t\leq\tau\), \(J_i=j\), and \(Z_i\), the transition is observed when \(C_i>t\). Thus the conditional expectation of its observation indicator divided by its weight denominator is \(P(C_i>t\mid Z_i)/G_{>}(t\mid Z_i)=1\).
Given \(T_i>\tau\) and \(Z_i\), the outcome is included when \(C_i>\tau\). Its observation indicator divided by \(G_{>}(\tau\mid Z_i)\) has conditional expectation \(P(C_i>\tau\mid Z_i)/G_{>}(\tau\mid Z_i)=1\).
Each possible category’s loss is multiplied by an observation indicator whose conditional expectation after weighting is one. Summing across categories gives \(E\{W_i(\tau)\mid T_i,J_i,Z_i\}=L_i(\tau)\), and hence the expected IPCW score equals the expected complete-data Brier score.
This is an expectation result, not a guarantee that a finite-sample weighted score equals the complete-data score. If \(G_{>}\) is estimated, the identity is approximate in finite samples and relies on a suitable censoring model for consistency.
Worked Example: A Three-Category Brier Loss
Suppose a subject has predicted probabilities \(p_A=0.30\), \(p_B=0.20\), and \(p_0=0.50\) for transitions A, B, and no transition by 12 months. Compute the complete-data loss for each possible observed category.
If the outcome is A, the loss is \((1-0.30)^2+(0-0.20)^2+(0-0.50)^2=0.49+0.04+0.25=0.78\). If the outcome is B, it is \((0-0.30)^2+(1-0.20)^2+(0-0.50)^2=0.09+0.64+0.25=0.98\). If there is no transition by 12 months, it is \((0-0.30)^2+(0-0.20)^2+(1-0.50)^2=0.09+0.04+0.25=0.38\).
These are the losses to use for the corresponding observed statuses. For example, an observed A transition at month 4 with \(G_{>}(4)=0.80\) contributes \(0.78/0.80=0.975\). The probability vector sums to one, as required, and the loss is computed against the same three categories used in the prediction.
Worked Example: Computing a Binary Transition-Specific IPCW Score
At a 6-month horizon, consider a single target transition A. For this binary score, \(Y_i=1\) if A occurs by 6 months and \(Y_i=0\) otherwise; a competing transition is a known \(Y_i=0\). Four hypothetical validation records have the following values:
| Subject | Prediction \(p_i\) | Observed record | Censoring survival used |
|---|---|---|---|
| 1 | 0.40 | A at month 2 | \(G_{>}(2)=0.80\) |
| 2 | 0.30 | Competing transition B at month 4 | \(G_{>}(4)=0.70\) |
| 3 | 0.20 | No transition observed through month 6; \(U_3>6\) | \(G_{>}(6)=0.60\) |
| 4 | 0.60 | Censored at month 3, with status at month 6 unknown | — |
Subject 1 has \(Y_1=1\), so the loss is \((1-0.40)^2=0.36\); the weighted contribution is \(0.36/0.80=0.4500\). Subject 2 has \(Y_2=0\), so its loss is \((0-0.30)^2=0.09\); its contribution is \(0.09/0.70=0.1286\). Subject 3 also has \(Y_3=0\), with loss \((0-0.20)^2=0.04\); its contribution is \(0.04/0.60=0.06667\). Subject 4 has no known horizon status and contributes no loss. Therefore,
The denominator is all four validation subjects, not just the three with known status. The example assumes the listed censoring probabilities apply to the relevant subjects and that censoring is conditionally independent of transition status given the variables used in the censoring model.
Endpoint Conventions Matter
The denominator must match the event in the numerator. Here \(I(U_i>\tau)\) requires \(C_i>\tau\), so its observation probability is \(P(C_i>\tau\mid Z_i)=G_{>}(\tau\mid Z_i)\). It is not generally \(P(C_i\geq\tau\mid Z_i)\). The distinction matters if censoring has a point mass at the horizon. A no-ties assumption between transition and censoring times does not rule out censoring exactly at \(\tau\).
Worked Example: A Censoring Atom at the Horizon
Suppose, for subjects with no transition by \(\tau\), that \(P(C<\tau)=0.10\), \(P(C=\tau)=0.20\), and \(P(C>\tau)=0.70\). The no-transition loss for a particular subject is \(L_0=0.38\). Under the strict rule, only \(C>\tau\) contributes, and the correct denominator is \(G_{>}(\tau)=0.70\). Conditional on that no-transition outcome, the expected weighted loss is \(0.70(0.38/0.70)=0.38\).
If the same strict indicator is incorrectly divided by \(P(C\geq\tau)=0.90\), the expected contribution is \(0.70(0.38/0.90)=0.2956\), which is too small. Subjects censored exactly at \(\tau\) are omitted by the strict indicator, so their probability must not be included in its denominator. A different endpoint convention can be used, but the indicator and its corresponding observation probability must be changed together.
Practical Checks, Alternatives, and Limitations
Before interpreting a score, check that the prediction categories are coherent and that the horizon and time origin match the observed transition records. Confirm that every event contribution uses the censoring probability at that event time, while the no-transition contribution uses the probability beyond the horizon. Also inspect whether the censoring model includes variables that explain differences in follow-up, and verify that estimated probabilities are positive for the observations being weighted. Very small probabilities can produce highly influential weights and an unstable score; detailed weight diagnostics are a separate issue.
Complete-case scoring is simpler but generally estimates performance among subjects whose outcomes are observed, not necessarily in the full validation population. Treating every subject censored before the horizon as event-free wrongly assigns an unknown outcome. IPCW avoids those two errors under its conditional independence and positivity assumptions, but it can be variable when censoring is heavy or weights are large. A marginal censoring survival estimate is appropriate only when censoring is independent without conditioning on covariates; otherwise, conditional modeling is needed. Model-based imputation of unknown statuses is another possible approach, but it relies on an outcome model rather than the censoring-weight identity derived here.
The method does not repair informative censoring that remains after conditioning on \(Z_i\), nor can it recover performance in covariate regions where the probability of remaining uncensored is zero. A high weighted score can also reflect poor predictions, unstable weights, or both; the score alone does not distinguish these explanations. Report the horizon, outcome definition, censoring model, endpoint convention, and whether the censoring survival function was estimated.
Check Your Understanding
Answer each question, then compare with the concise solutions below.
- For a known transition of type \(j\) at time \(t\leq\tau\), what censoring survival probability belongs in the IPCW denominator under the stated convention?
- Why does a subject censored before \(\tau\), without an earlier observed transition, contribute no direct loss?
- If \(G_{>}(\tau)=0.50\), a subject is observed event-free beyond \(\tau\), and their no-transition loss is \(0.12\), what is their weighted contribution?
- What assumption permits the observation probability for a no-transition outcome to be written using only the censoring distribution conditional on \(Z\)?
- Suppose censoring occurs exactly at \(\tau\) with positive probability. Why is \(P(C\geq\tau\mid Z)\) not the correct denominator for the indicator \(I(U>\tau)\)?
Solutions. (1) Use \(G_{>}(t\mid Z)=P(C>t\mid Z)\), the probability censoring occurs after the event time. (2) The subject’s transition status at \(\tau\) is unknown, so a complete-data loss cannot be assigned; IPCW compensates through the observed subjects’ weights. (3) The contribution is \(0.12/0.50=0.24\). (4) Conditional independence of censoring and transition time/type given \(Z\). (5) The strict indicator requires \(C>\tau\); the probability \(P(C\geq\tau\mid Z)\) also includes censoring exactly at \(\tau\), which is excluded by that indicator.