Transition-Specific Prediction Validation · Tutorial 402 of 699

Integrated Brier Scores Across Transition Horizons

You will learn to define an integrated transition-specific Brier score, compute it from pointwise scores, and explain how the chosen horizon weights shape comparisons between prediction models.

Advanced 12 min read

What You'll Learn

Define an integrated transition-specific Brier score over a prespecified time interval.
Distinguish uniform time weighting from other horizon-weighting schemes.
Approximate the integral from pointwise scores using the trapezoidal rule.
Interpret the score on its squared-probability-error scale.
Diagnose how interval choice, censoring adjustment, and time weights affect comparisons.
Recognize when different horizon weights can reverse a model ranking.

Why Integrate Scores Across Horizons?

A transition-specific prediction can be evaluated at several horizons, but a model that performs well at one time may perform less well at another. Reporting only one horizon gives a focused answer for that time; it does not summarize performance across a time range. An integrated Brier score aggregates pointwise Brier scores over a prespecified interval, while making explicit how much weight each part of that interval receives.

The integration is not merely a numerical convenience. The chosen weights define the summary being reported. Uniform weighting treats equal lengths of time equally. A different weighting scheme can emphasize early or late predictions and can therefore change which model has the smaller score. Since smaller Brier scores indicate better average prediction accuracy, the weighting rule is part of the scientific question, not a cosmetic choice.

Definition: Let $BS_k(t)$ denote the population transition-specific Brier score at horizon $t$ for transition $k$, using the prediction target defined for that transition. Over a prespecified interval $[a,b]$, where $a<b$, the integrated Brier score with weighting density $w(t)$ is $$ IBS_k^{w}(a,b)=\int_a^b w(t)BS_k(t)\,dt, \qquad w(t)\geq 0,\qquad \int_a^b w(t)\,dt=1. $$ The uniform-weight version has $w(t)=1/(b-a)$, so $$ IBS_k^{\mathrm{unif}}(a,b)=\frac{1}{b-a}\int_a^b BS_k(t)\,dt. $$

At a fixed horizon, the Brier score is the expected squared difference between the binary outcome for the specified transition and its predicted probability. In a validation sample of $n$ subjects with fully observed outcomes, its empirical form is $n^{-1}\sum_{i=1}^n\{Y_{ik}(t)-\hat p_{ik}(t)\}^2$. Here $Y_{ik}(t)$ is the transition-specific binary outcome and $\hat p_{ik}(t)$ is the corresponding predicted probability. When a competing outgoing transition is a non-target outcome under the validation definition, it is coded as such; it is not treated as censoring.

With right censoring, the pointwise score must be estimated using an appropriate censoring-adjusted method, such as inverse-probability-of-censoring weighting. Integration does not repair a biased pointwise estimator. The censoring adjustment must be valid at the horizons being summarized; in particular, its conditional independent-censoring assumptions and positivity condition must be reasonable, with non-negligible probability of remaining uncensored long enough to determine the relevant outcomes. The prediction target and validation population must also be defined consistently across the interval.

What the Weighting Means

The uniform integrated score is the average pointwise score over time, not the unscaled area under the score curve. Dividing by $b-a$ gives a time average and keeps the result on the Brier-score scale. A general weighting density $w(t)$ also produces a weighted average, but it changes the relative contribution of horizons. Because $w(t)$ has units of inverse time, its product with $dt$ is dimensionless; the resulting integrated score remains on the squared-probability-error scale.

For a finite grid $a=t_0<t_1<\cdots<t_m=b$, pointwise estimated scores are usually integrated numerically. Under uniform weighting, the trapezoidal approximation is $$ \widehat{IBS}_k^{\mathrm{unif}}(a,b) = \frac{1}{b-a} \sum_{j=0}^{m-1} (t_{j+1}-t_j) \frac{\widehat{BS}_k(t_j)+\widehat{BS}_k(t_{j+1})}{2}. $$

This formula approximates the area by trapezoids and then divides by the interval length. Unequally spaced horizons contribute in proportion to their time gaps; simply averaging the listed values would give the same influence to every grid point, not every unit of time. For general weights, integrate $w(t)BS_k(t)$. A practical approximation can use trapezoids for that product, or an explicitly prespecified quadrature rule. The grid and numerical rule should be adequate to represent meaningful changes in the score curve.

Worked Example: Uniform Integration on an Uneven Grid

Suppose a hypothetical validation analysis estimates a transition-specific Brier score of 0.080 at year 0, 0.100 at year 1, and 0.070 at year 3. Assume these are valid pointwise estimates for the same transition target and validation population, including any required censoring adjustment. Find the uniform integrated score on $[0,3]$.

The intervals have lengths 1 and 2 years. Their trapezoidal areas are $$ 1\left(\frac{0.080+0.100}{2}\right)=0.090, \qquad 2\left(\frac{0.100+0.070}{2}\right)=0.170. $$

The total estimated area is $0.090+0.170=0.260$ score-years. Dividing by the 3-year interval gives $$ \widehat{IBS}^{\mathrm{unif}}(0,3)=\frac{0.260}{3}=0.0867. $$

Thus the model's average estimated Brier score across years 0 to 3 is 0.0867 under uniform time weighting. It is not the unscaled area 0.260, and it is not the unweighted mean of the three grid values, which would be 0.0833. The latter incorrectly gives the single year-1 grid point as much influence as the entire two-year segment from year 1 to year 3.

How Weighting Can Change a Model Comparison

For two models evaluated on the same transition target, interval, and validation population, a smaller integrated score means smaller weighted average squared probability error over that interval. It does not imply that the model is better at every horizon. In particular, models whose pointwise score curves cross can trade places when the weighting changes.

Worked Example: A Prespecified Early-Horizon Emphasis

Suppose, hypothetically, that model M has estimated pointwise scores 0.080, 0.100, and 0.120 at years 0, 1, and 3, respectively. Model N has scores 0.100, 0.090, and 0.070 at those horizons. Assume valid pointwise estimates and linear interpolation between adjacent grid points. First calculate the uniform integrated scores over $[0,3]$.

For M, the trapezoidal area is $1(0.080+0.100)/2+2(0.100+0.120)/2=0.090+0.220=0.310$, so its uniform integrated score is $0.310/3=0.1033$. For N, the area is $1(0.100+0.090)/2+2(0.090+0.070)/2=0.095+0.160=0.255$, giving $0.255/3=0.0850$. Uniform weighting favors N.

Now suppose the scientific aim, specified before examining these results, is to emphasize the first year. Assign 75% of total weight uniformly to $[0,1]$ and 25% uniformly to $[1,3]$. The resulting weighted score is 0.75 times the average score on $[0,1]$, plus 0.25 times the average score on $[1,3]$. For M those interval averages are 0.090 and 0.110, giving $0.75(0.090)+0.25(0.110)=0.0950$. For N they are 0.095 and 0.080, giving $0.75(0.095)+0.25(0.080)=0.09125$. N still has the smaller score, but the difference is reduced from 0.0183 under uniform weighting to 0.00375 under the early-emphasis weighting.

The weighting changed the size of the comparison because M is relatively better late in the interval, where the early-emphasis scheme assigns less weight. The calculation does not show that one weighting is universally correct: the appropriate choice depends on the intended prediction use and should be justified in advance.

Worked Example: A Ranking Reversal From Late-Horizon Weighting

Consider two other hypothetical models on $[0,2]$, with pointwise scores at years 0, 1, and 2. Model A has scores 0.120, 0.130, and 0.000; model B has scores 0.070, 0.080, and 0.090. Assume valid pointwise estimates and linear interpolation between horizons.

Under uniform weighting, A's area is $(0.120+0.130)/2+(0.130+0.000)/2=0.125+0.065=0.190$, so its integrated score is $0.190/2=0.0950$. B's area is $(0.070+0.080)/2+(0.080+0.090)/2=0.075+0.085=0.160$, giving an integrated score of $0.160/2=0.0800$. Uniform weighting favors B.

Now place 25% of total weight uniformly on $[0,1]$ and 75% uniformly on $[1,2]$. The interval-average scores are 0.125 and 0.065 for A, and 0.075 and 0.085 for B. The weighted scores are $0.25(0.125)+0.75(0.065)=0.0800$ for A and $0.25(0.075)+0.75(0.085)=0.0825$ for B. This late-emphasis weighting favors A. The ranking reversal is mathematically consistent: A's score is much smaller late, while B's is smaller early. The example also illustrates why a weighting scheme should be selected for its substantive relevance, rather than chosen after seeing which model it favors.

Calculation, Diagnostics, and Limitations

A defensible integrated-score analysis requires decisions about the target transition, prediction origin, interval, weighting rule, pointwise estimation method, and numerical integration rule. The interval should lie within the range where predictions and outcome information support evaluation. If censoring-adjusted scores become unstable near the end of follow-up because few subjects remain observable, integrating them can retain that instability rather than remove it.

Useful checks include plotting the pointwise score estimates before summarizing them, inspecting the number at risk or the effective information supporting late horizons, and checking censoring weights for extreme values. Also verify that the evaluation grid covers the prespecified interval, that interval lengths are used correctly, and that numerical integration uses the intended weighting density. A sharply changing curve may require a denser grid; an apparently smooth curve does not establish that the censoring model or pointwise estimator is valid.

The integrated Brier score has important limitations. A single number hides horizon-specific patterns, and two models with similar integrated scores can have different pointwise performance. A result is conditional on the target definition, prediction origin, interval, weighting rule, validation population, and censoring adjustment. It should therefore be accompanied by those choices and, where useful, the pointwise score curve. The score evaluates probabilistic accuracy through squared error; it does not by itself explain calibration, discrimination, or the clinical consequences of particular errors. Those questions call for complementary summaries rather than an interpretation beyond what the integrated score measures.

Common Mistakes

Averaging grid values without regard to spacing. This treats each supplied horizon equally, even when the gaps differ. For a time average, integrate over time and divide by the interval length, using interval widths in the calculation.
Reporting area as the uniform integrated score. The area has score-times-time units. Divide it by $b-a$ to obtain the uniform average score.
Calling the result a horizon-free property of a model. The integrated score depends on the interval and weights. Report them as part of the result.
Assuming a smaller integrated score means improvement at every horizon. It only indicates a smaller weighted average. Inspect pointwise scores to locate where performance differs.
Assuming integration fixes censoring problems. Integration combines pointwise estimates; it cannot correct invalid censoring assumptions, poor positivity, or unstable weights.

Key takeaway: An integrated transition-specific Brier score is a weighted average of pointwise scores over a prespecified interval. The interval, time weights, pointwise censoring adjustment, and numerical integration rule jointly determine what the summary means.

Check Your Understanding

Answer each question, then compare with the solutions below.

Pointwise scores are 0.060, 0.100, and 0.080 at years 0, 1, and 2. Under uniform weighting and linear interpolation, calculate the integrated score on $[0,2]$.
Why can an unweighted arithmetic mean of pointwise scores be inappropriate when the horizons are unequally spaced?
Two models have crossing pointwise score curves. Can their ranking depend on the weighting density? Explain.
Name two checks relevant to the reliability of a late-horizon integrated score under right censoring.

Solutions. (1) The trapezoidal area is $(0.060+0.100)/2+(0.100+0.080)/2=0.080+0.090=0.170$. Divide by 2 years: the integrated score is $0.170/2=0.0850$. (2) Equal weight per listed point ignores how much time each point represents; the time integral uses the grid's interval widths. (3) Yes. Different densities assign different relative weight to times at which each model performs better, so the weighted averages, and potentially the ranking, can change. (4) Check whether censoring weights are extreme and whether enough subjects remain observable to support the late-horizon estimates; also assess the censoring model and its positivity assumption.

Back to Tutorials