Inverse-Weighted Brier Scores With Censoring

Why censoring changes the Brier score

At a fixed horizon $t$, a survival prediction can be expressed as an event risk $r_i(t)$: the predicted probability that individual $i$ experiences the event by time $t$. The Brier score compares that prediction with the eventual binary outcome $I(T_i\leq t)$. A prediction close to 1 is good when the event occurs by $t$; a prediction close to 0 is good when the person remains event-free through $t$.

Censoring makes that outcome unavailable for some individuals. Someone censored before $t$ may or may not experience the event by $t$, so assigning that person either outcome is unjustified. Simply dropping censored people can also bias the score: those with observed outcomes may differ systematically from those whose outcomes are censored. Inverse-probability-of-censoring weighting (IPCW) addresses this by giving more weight to observed outcomes that were less likely to remain observable.

Definition: Let $T_i$ be the event time, $C_i$ the censoring time, $U_i=\min(T_i,C_i)$ the observed follow-up time, and $\delta_i=I(T_i\leq C_i)$ the event indicator. The target Brier score at horizon $t$ is $$ BS(t)=E\left[\{I(T\leq t)-r(t)\}^2\right]. $$ An event observed by $t$ has outcome 1; an individual known to be event-free beyond $t$ has outcome 0. Define $G(s\mid Z)=P(C>s\mid Z)$, the conditional probability of remaining uncensored beyond $s$, given covariates $Z$. For an observed event at time $u$, use the left limit $G(u-\mid Z)=P(C\geq u\mid Z)$. We assume continuous event and censoring times or a consistent convention for ties.

The prediction $r_i(t)$ is treated as fixed for evaluation; it may have been produced by a joint model. The covariates $Z_i$ include the information needed to model censoring and the information on which the prediction depends. At a fixed horizon, individuals with $U_i>t$ are known non-cases by $t$. Those censored before $t$ without an earlier event have an unknown outcome and contribute no direct term.

The IPCW score and its conditions

An observed event by $t$ contributes its squared error for outcome 1, namely $\{1-r_i(t)\}^2$. Its weight is the inverse probability that censoring did not occur before the event. An individual observed beyond $t$ contributes the squared error for outcome 0, $r_i(t)^2$, weighted by the inverse probability of remaining uncensored through $t$. The resulting estimator is

\widehat{BS}_{\mathrm{IPCW}}(t) = \frac{1}{n}\sum_{i=1}^{n} \left[ \frac{I(U_i\leq t,\delta_i=1)\{1-r_i(t)\}^2} {\widehat G(U_i-\mid Z_i)} + \frac{I(U_i>t)r_i(t)^2} {\widehat G(t\mid Z_i)} \right].

Here $\widehat G$ is a known or estimated censoring survival function. The denominator is the same probability as the relevant observation event: seeing an event at its time requires avoiding censoring until that time, while knowing someone is event-free beyond $t$ requires avoiding censoring through $t$. Individuals censored before $t$ without an observed event have both indicators equal to zero.

Conditions for unbiasedness: For the population score to be recovered in expectation by the IPCW expression with the true $G$, censoring must be conditionally independent of the event time given $Z$, the censoring survival probabilities must be correctly specified, and the relevant probabilities must be positive. In particular, $G(T-\mid Z)>0$ for event times up to $t$ and $G(t\mid Z)>0$ in the target population. With a fixed prediction rule and known $G$, these conditions make the sample average unbiased for $BS(t)$ under independent sampling. Replacing $G$ with an estimated model generally does not give exact finite-sample unbiasedness, though consistent estimation can support consistency of the score estimate.

Why the weights recover the complete-outcome loss

The justification can be seen by considering censoring conditional on the event time and $Z$. Write $L_i(t)=\{I(T_i\leq t)-r_i(t)\}^2$ for the complete-outcome loss. If $T_i\leq t$, the event is observed when $C_i\geq T_i$. Under conditional independence, the probability of this observation is $G(T_i-\mid Z_i)$. Therefore the conditional expected weighted event contribution is

E\left[ \left. \frac{I(U_i\leq t,\delta_i=1)\{1-r_i(t)\}^2} {G(U_i-\mid Z_i)} \right|T_i,Z_i \right] = \{1-r_i(t)\}^2 = L_i(t),\qquad T_i\leq t.

If $T_i>t$, the person is known to be event-free at $t$ when $C_i>t$. That occurs with probability $G(t\mid Z_i)$. Hence

E\left[ \left. \frac{I(U_i>t)r_i(t)^2}{G(t\mid Z_i)} \right|T_i,Z_i \right] = r_i(t)^2 = L_i(t),\qquad T_i>t.

Taking expectations over $T_i$ and $Z_i$, then averaging across individuals, gives $E\{\widehat{BS}_{\mathrm{IPCW}}(t)\}=BS(t)$ when the true censoring probabilities are used and the assumptions hold. This argument also explains positivity: division by a zero probability is undefined, and very small probabilities produce highly variable contributions.

Worked Example: Calculating the Score at 12 Months

Consider five hypothetical validation participants and a fixed horizon of $t=12$ months. Their predicted risks, observed follow-up, and censoring probabilities are shown below. For simplicity, treat the listed censoring probabilities as known.

Person	$r_i(12)$	Observed information	Relevant $G$
A	0.20	Event at month 4	$G(4-\mid Z_A)=0.80$
B	0.50	Event at month 9	$G(9-\mid Z_B)=0.60$
C	0.30	Observed event-free beyond month 12	$G(12\mid Z_C)=0.50$
D	0.40	Censored at month 7 without an event	Not used
E	0.70	Event at month 11	$G(11-\mid Z_E)=0.55$

A's event contribution is $(1-0.20)^2/0.80=0.64/0.80=0.800$. B's is $(1-0.50)^2/0.60=0.25/0.60=0.4167$. C's known non-case contribution is $0.30^2/0.50=0.09/0.50=0.180$. E's event contribution is $(1-0.70)^2/0.55=0.09/0.55=0.1636$. D's outcome by month 12 is unknown, so both indicators for D are zero.

The estimate divides the sum of the four contributions by the full sample size $n=5$, not by the number with observed outcomes: $$ \widehat{BS}_{\mathrm{IPCW}}(12) = \frac{0.800+0.4167+0.180+0.1636}{5} = 0.3121. $$ This is an estimated mean squared error on the binary-outcome scale. Lower values indicate better predictions for this horizon; the score alone does not identify whether any error arises from calibration or discrimination.

Worked Example: Checking Which Squared Error Belongs in Each Term

Suppose a hypothetical participant has predicted event risk $r_i(t)=0.20$. If an event is observed before $t$, the outcome is 1, so the unweighted loss is $(1-0.20)^2=0.64$. If instead the participant is known to remain event-free beyond $t$, the outcome is 0 and the unweighted loss is $(0-0.20)^2=0.04$.

If the relevant censoring probability is 0.80 in either case, the corresponding weighted contributions are $0.64/0.80=0.800$ for the event and $0.04/0.80=0.050$ for the known non-case. The event term must use $\{1-r_i(t)\}^2$, and the known non-case term must use $r_i(t)^2$. Reversing these terms would reward a low event-risk prediction when an event actually occurred, rather than penalizing its error.

Diagnostics, alternatives, and limitations

A correct-looking formula does not guarantee a reliable estimate. Inspect the fitted censoring survival probabilities over the time range that contributes to the score, especially near $t$ and at observed event times. Check whether the censoring model includes variables associated with both censoring and event follow-up, and whether predicted censoring probabilities agree with observed censoring patterns across relevant covariate groups. A model that assigns near-zero probabilities to observed cases creates extreme weights.

One useful warning signal is the effective sample size of a set of weights $w_i$, calculated as $(\sum_i w_i)^2/\sum_i w_i^2$. A small value relative to the number of contributing observations indicates that a few individuals dominate. The event and non-case terms use different weights, so inspect their weight distributions separately. In small samples, a finite-sample IPCW estimate can exceed 1 even though the population Brier score lies between 0 and 1: the inverse weights can make individual contributions large. This is a warning about variability or weak positivity, not by itself proof of an arithmetic error. Truncating weights may reduce variability, but changes the estimator and generally sacrifices the unbiasedness justification above.

Worked Example: Recognizing a Positivity Warning

At a fixed horizon, suppose a hypothetical sample has $n=3$. One participant has an event by $t$, predicted risk 0.10, and $G(T-\mid Z)=0.10$. A second is observed event-free beyond $t$, predicted risk 0.20, and $G(t\mid Z)=0.10$. The third is censored before $t$ without an event.

The event contribution is $(1-0.10)^2/0.10=0.81/0.10=8.10$. The known non-case contribution is $0.20^2/0.10=0.04/0.10=0.400$. The censored participant contributes zero directly, so $$ \widehat{BS}_{\mathrm{IPCW}}(t) = \frac{8.10+0.400}{3} = 2.833. $$

This value is far above the possible range of the target population score, which is from 0 to 1. The calculation is correct, but the tiny censoring probabilities make the estimate extremely unstable in this sample. Investigators should examine the censoring model, the observations receiving large weights, and whether the target horizon is supported by adequate follow-up; they should not silently report the result as a stable performance estimate.

The main alternatives have different risks. An unweighted complete-case score averages only among people whose outcome is known by $t$; it can target a selected group if censoring depends on prognosis. Treating everyone censored before $t$ as event-free assigns an outcome that was not observed and generally biases the score. IPCW uses the full cohort denominator and adjusts observed losses for their probability of being observed, but it relies on conditional independent censoring, a suitable censoring model, and positivity. Other model-based strategies can address missing outcomes under their own assumptions; none removes the need to state how censoring is handled.

Common pitfall: “A participant was censored before the horizon, so count that person as event-free and use $r_i(t)^2$.” This is wrong because censoring before $t$ does not establish that $T_i>t$. The correction is to omit that person’s direct contribution and use IPCW for observed event and known event-free outcomes, under the stated assumptions.

Check Your Understanding

Try each question, then compare your work with the solutions below.

At $t=10$, a person has predicted event risk 0.30 and an observed event at month 6. If $G(6-\mid Z)=0.75$, what is the weighted contribution?
A person has risk 0.40 and is observed event-free beyond $t$, with $G(t\mid Z)=0.50$. Calculate the contribution.
Why is a person censored at month 5 without an event not scored as an event-free individual at $t=10$?
Name two conditions required for the IPCW score using the true $G$ to recover the target Brier score in expectation.
What does an estimated score above 1 suggest, and does it necessarily imply a computational mistake?

Solutions. (1) The event loss is $(1-0.30)^2=0.49$, so the weighted contribution is $0.49/0.75=0.6533$. (2) The non-case loss is $0.40^2=0.16$, giving $0.16/0.50=0.320$. (3) Follow-up ending before month 10 leaves the event status by month 10 unknown; the person is not known to have $T>10$. (4) Examples include conditional independence of censoring and event time given $Z$, correct specification of the censoring survival function, and positivity of the relevant censoring probabilities. (5) Extreme weights or finite-sample variability may produce a score above 1. Since the population score is bounded by 1, this warrants diagnostic checks, but it does not alone prove an arithmetic error.

Person	\(r_i(12)\)	Observed information	Relevant \(G\)
A	0.20	Event at month 4	\(G(4-\mid Z_A)=0.80\)
B	0.50	Event at month 9	\(G(9-\mid Z_B)=0.60\)
C	0.30	Observed event-free beyond month 12	\(G(12\mid Z_C)=0.50\)
D	0.40	Censored at month 7 without an event	Not used
E	0.70	Event at month 11	\(G(11-\mid Z_E)=0.55\)

Inverse-Weighted Brier Scores With Censoring

What You'll Learn