The Integrated Brier Score in Joint Models

Why Integrate Brier Scores Across Time?

A survival prediction is rarely used at only one horizon. A joint model may provide a patient's predicted survival probability at several future times, conditional on the longitudinal history available at a prediction landmark. The Brier score assesses the accuracy of the survival prediction at one chosen horizon; integrating those scores summarizes accuracy over a range of horizons. That summary is useful only when the range and any time weights match the intended prediction task.

For example, an average over the next two years answers a different question from an average over the next ten years. A weighted average that emphasizes near-term horizons answers a different question again. The integrated Brier score is therefore not a horizon-free property of a fitted model: it is a summary defined relative to the prediction setting, interval, and weighting rule.

Definition and Censoring Adjustment

Let $T_i$ be the event time for validation subject $i$, and $C_i$ the censoring time. The observed time is $\tilde T_i=\min(T_i,C_i)$, and $\Delta_i=I(T_i\le C_i)$ indicates whether the event was observed. At a prediction landmark, let $\hat S_i(t)$ be the joint model's predicted probability that subject $i$ remains event-free for at least $t$ time units after that landmark. The binary outcome at horizon $t$ is $I(T_i>t)$. Thus the prediction error for a subject whose outcome is known is the squared difference between this binary outcome and $\hat S_i(t)$.

Definition: The population Brier score at horizon $t$ is $BS(t)=E\{[I(T>t)-\hat S(t)]^2\}$. The integrated Brier score over a prespecified interval $[\tau_1,\tau_2]$, with $\tau_1<\tau_2$, is the time average of the horizon-specific scores: $$ IBS(\tau_1,\tau_2)=\frac{1}{\tau_2-\tau_1}\int_{\tau_1}^{\tau_2} BS(t)\,dt. $$ Lower values indicate smaller average squared prediction error over that interval; the score is not itself a measure of discrimination alone.

With right censoring, the outcome at horizon $t$ is unknown for a subject censored before $t$. Inverse-probability-of-censoring weighting (IPCW), covered in the earlier material on inverse-weighted Brier scores, gives greater weight to comparable observed outcomes when censoring has made them less likely to be observed. Let $G(u\mid Z_i)=P(C_i\ge u\mid Z_i)$, where $Z_i$ denotes variables used in the censoring model. A common estimated horizon-specific score is

\widehat{BS}(t)=\frac{1}{n}\sum_{i=1}^n\left[ \frac{I(\tilde T_i\le t,\Delta_i=1)}{\hat G(\tilde T_i^-\mid Z_i)} \{0-\hat S_i(t)\}^2 + \frac{I(\tilde T_i>t)}{\hat G(t\mid Z_i)} \{1-\hat S_i(t)\}^2 \right].

The first term uses subjects with an observed event by $t$; their event-free outcome is zero. The second uses subjects known to be event-free through $t$; their outcome is one. A subject censored before $t$ contributes neither term at that horizon. The left limit in $\hat G(\tilde T_i^-\mid Z_i)$ handles censoring survival immediately before an observed event time. Equivalent formulations use predicted event risk $1-\hat S_i(t)$ and the event indicator; the squared errors are the same.

Assumptions and Interpretation

The IPCW score estimates the desired prediction error only under conditions that make the observed outcomes representative after weighting. In particular, censoring must be independent of the event time conditional on the variables $Z_i$ used in the censoring model, or a suitable censoring model must otherwise correctly account for the dependence. The relevant censoring probabilities must be positive over the interval: if $G(t\mid Z_i)=0$, an outcome at $t$ cannot be recovered by weighting. Event and censoring times must be measured consistently, including a clear convention for ties.

The predictions must also correspond to a clearly defined validation task. In a joint-model setting, this includes specifying the prediction landmark and which longitudinal measurements are available at that landmark. The same rules for obtaining predictions should be applied throughout the validation set. Performance estimated on data used to fit or tune the model can be optimistic; the score's definition does not by itself correct that problem.

The integration interval should be prespecified and restricted to horizons where predictions and outcomes are meaningfully estimable. For a long horizon with very few subjects still known to be event-free, IPCW weights can become large and the estimated score unstable. A finite interval is not a remedy for unsupported horizons: the censoring distribution and effective information still need to be examined.

Computing the Integral From Horizon-Specific Scores

In practice, scores are often calculated at a grid of horizons. If the grid is $t_1<\cdots<t_K$, the trapezoidal rule approximates the area between adjacent score values by a trapezoid. Dividing the resulting area by the interval width gives the time-average score. The grid should be sufficiently dense where the score curve changes rapidly, and it should include the prespecified interval's endpoints or use a stated interpolation rule to handle them.

\widehat{IBS}(\tau_1,\tau_2) \approx \frac{1}{\tau_2-\tau_1} \sum_{k=1}^{K-1} (t_{k+1}-t_k)\frac{\widehat{BS}(t_k)+\widehat{BS}(t_{k+1})}{2}.

This is a numerical approximation, not a new horizon-specific score. If scores are available only at widely separated horizons, the approximation implicitly assumes a simple shape between them. The area before normalization has units of time, while the normalized IBS has the same dimensionless scale as the Brier score.

Worked Example: A Censoring-Adjusted Brier Score at One Horizon

Consider four hypothetical validation subjects and a horizon of $t=2$ years after the prediction landmark. Subject 1 has an observed event at 1 year and $\hat S_1(2)=0.70$, with $\hat G(1^-)=0.90$. Subjects 2 and 4 are known to be event-free beyond 2 years, with predicted survival probabilities $0.80$ and $0.60$, respectively; both have $\hat G(2)=0.80$. Subject 3 is censored at 1.5 years, before the horizon, so the subject's event status at 2 years is unknown. Assume these censoring probabilities are the applicable estimates for this hypothetical calculation.

Subject 1 contributes an event outcome of zero. Its weighted squared error is $(0-0.70)^2/0.90=0.49/0.90=0.5444$. Subject 2 contributes $(1-0.80)^2/0.80=0.04/0.80=0.0500$. Subject 4 contributes $(1-0.60)^2/0.80=0.16/0.80=0.2000$. Subject 3 contributes zero at this horizon because censoring occurred before the outcome could be determined; this does not mean the subject's prediction error is known to be zero.

The IPCW estimate divides the sum of the four subject-level terms by $n=4$: $$ \widehat{BS}(2)=\frac{0.5444+0.0500+0+0.2000}{4}=0.1986. $$ The result is an estimated squared prediction error at 2 years, not an average over time. In particular, the event contribution uses one observed event, and the two event-free contributions use subjects 2 and 4.

Worked Example: Trapezoidal Integration Over a Prespecified Interval

Suppose the estimated horizon-specific Brier scores in a hypothetical validation are $0.18$ at year 1, $0.22$ at year 2, and $0.20$ at year 4. The prespecified interval is $[1,4]$ years. Using the trapezoidal rule, the area from year 1 to year 2 is $(2-1)(0.18+0.22)/2=0.20$ score-years. The area from year 2 to year 4 is $(4-2)(0.22+0.20)/2=0.42$ score-years.

The approximate total area is $0.20+0.42=0.62$ score-years. The interval width is $4-1=3$ years, so $$ \widehat{IBS}(1,4)=\frac{0.62}{3}=0.2067. $$ This is the average estimated Brier score across years 1 through 4, under the linear interpolation between the supplied grid values. It is not the score at year 2, and it does not assign equal weight to the three reported grid points: the interval from year 2 to year 4 is twice as long as the first interval.

Time Weighting Changes the Question

The ordinary IBS gives each unit of time equal weight. A different prespecified weighting function $w(t)$, with $w(t)\ge 0$ and a finite positive integral over the interval, defines a weighted time average:

IBS_w(\tau_1,\tau_2)= \frac{\int_{\tau_1}^{\tau_2} w(t)BS(t)\,dt} {\int_{\tau_1}^{\tau_2} w(t)\,dt}.

Because the denominator normalizes the weights, multiplying $w(t)$ everywhere by the same positive constant does not change the result. A weight concentrated near the landmark emphasizes near-term prediction accuracy; a weight concentrated later emphasizes longer-term accuracy. The weighting rule should be chosen for the prediction question, not selected after inspecting which version gives a more favorable model score.

Worked Example: Comparing Equal and Unequal Time Weights

Suppose a hypothetical Brier-score curve is approximated by straight lines between $BS(1)=0.10$, $BS(3)=0.20$, and $BS(5)=0.30$. Over $[1,5]$, the unweighted area from year 1 to year 3 is $2(0.10+0.20)/2=0.30$, and the area from year 3 to year 5 is $2(0.20+0.30)/2=0.50$. Thus the unweighted IBS is $(0.30+0.50)/4=0.2000$.

Now set $w(t)=2$ on $[1,3)$ and $w(t)=1$ on $[3,5]$. The weighted area is $2(0.30)+1(0.50)=1.10$, and the total weight over time is $2(2)+1(2)=6$. Therefore, $$ IBS_w(1,5)=\frac{1.10}{6}=0.1833. $$ The weighted score is lower here because the earlier interval, where the hypothetical scores are lower, receives twice the weight per year. This difference reflects a changed summary target, not an improvement in the predictions.

Justification and Practical Checks

The integrated score follows directly from averaging squared prediction error over time. At each $t$, the target is the binary event-free indicator and the prediction is its estimated probability. The expected squared difference is $BS(t)$. Averaging this quantity over a time interval gives the area under the Brier-score curve divided by the interval width. For nonnegative, integrable scores, this can be expressed as an average over time and validation subjects:

\frac{1}{\tau_2-\tau_1}\int_{\tau_1}^{\tau_2} E\{[I(T>t)-\hat S(t)]^2\}\,dt = E\left\{\frac{1}{\tau_2-\tau_1}\int_{\tau_1}^{\tau_2} [I(T>t)-\hat S(t)]^2\,dt\right\}.

The equality is justified by interchanging expectation and integration for an integrable squared error. Censoring prevents direct observation of all the terms on the right, so IPCW is used to estimate each horizon-specific expectation before integration. The integration itself does not remove bias from a misspecified censoring model or recover information at horizons with no censoring support.

Several diagnostics are important before interpreting an IBS:

Inspect the horizon-specific score curve. A single summary can conceal that performance is good early and poor later, or the reverse.
Check censoring weights and support. Very small estimated censoring survival probabilities can produce influential contributions and unstable scores.
Check the time grid and endpoints. Sparse horizons, especially around sharp changes in event incidence or score, can make trapezoidal integration inaccurate.
Confirm the prediction target. Landmark, available longitudinal history, time origin, and event definition must match the intended use.
Compare scores on the same interval and with the same weighting rule. Otherwise, differences may reflect distinct summaries rather than different predictive accuracy.

A fixed-horizon Brier score is preferable when a decision is tied to one clinically relevant time point and detail across time is unnecessary. The integrated score is useful when a range matters, but it can obscure time-specific weaknesses. Discrimination summaries such as time-dependent AUC answer a different question, emphasizing ranking rather than the squared accuracy of predicted probabilities. The logarithmic score is another scoring approach, but like any alternative it targets its own loss function and must be interpreted accordingly.

Common Pitfall and Limitations

Common mistake: “The IBS is a single overall property of the joint model, so two IBS values can be compared even if they use different intervals or weights.” This is incorrect. The interval and weighting rule define which time-specific errors are averaged. Compare values as the same summary only when their prediction target, validation setting, interval, and weighting are aligned.

The IBS also does not diagnose why predictions are inaccurate. A poor value can reflect calibration error, weak discrimination, or both; the score alone does not separate those sources. Conversely, a favorable average need not imply good predictions at every horizon. Results can be sensitive to censoring-model specification and sparse late follow-up, and the numerical integral depends on the score grid and interpolation. Report the interval, units, weighting rule, censoring adjustment, and horizon-specific pattern alongside the summary.

Check Your Understanding

Answer each question, then compare with the brief solutions.

Why does a subject censored before horizon $t$ generally contribute neither an event nor an event-free outcome term to the IPCW Brier score at $t$?
Hypothetical scores are $0.10$ at year 1 and $0.30$ at year 3. Using linear interpolation, calculate the unweighted IBS on $[1,3]$.
What does it mean if a weighted IBS puts twice as much weight per year on horizons from 1 to 2 years as on horizons from 2 to 4 years?
Name two conditions that make an IPCW integrated Brier score unreliable at late horizons.

Solutions. (1) Censoring before $t$ leaves the event status at $t$ unknown, so neither binary outcome is observed for that subject; IPCW reweights outcomes that are observed rather than treating the unknown status as an outcome. (2) The trapezoidal area is $ (3-1)(0.10+0.30)/2=0.40$ score-years, and dividing by the two-year interval gives $0.40/2=0.2000$. (3) The summary emphasizes near-term accuracy relative to accuracy at later times; it is a different target from an equally weighted time average. (4) Examples include very small estimated censoring survival probabilities, failure of conditional independent censoring, or too few subjects with observed outcome information to support the horizon.