Aggregating Performance Across Dynamic Updates

Why Aggregation Across Updates Needs a Target

A dynamic prediction is made at a particular update time, using the information available by then, for an outcome over a specified future horizon. Validation therefore produces a sequence of performance estimates, not necessarily one estimate for one fixed set of people. At later updates, some subjects may have experienced the event, been censored, or otherwise become ineligible; the prediction horizon may also differ. A single summary can be useful, but only after specifying which updates, which eligible populations, which horizons, and what weighting the summary represents.

Let \(i\) index subjects and \(k\) index update times \(s_k\). At update \(k\), let \(E_k\) be the set of eligible subjects, with size \(n_k\). Eligibility is defined by the prediction question; for example, a prediction of event risk over the next \(h_k\) time units generally concerns subjects still event-free at \(s_k\). Let \(M_k\) denote a validation metric estimated among \(E_k\), using predictions and outcomes defined for that update and horizon. The metric could be a loss, such as a Brier score, or a measure such as the area under the receiver operating characteristic curve (AUC).

Definition: An update-specific metric is a performance measure calculated for predictions made at one update time, for a stated eligible population and prediction horizon. An aggregate metric combines update-specific metrics using explicit weights. Those weights define how much each update-specific target contributes to the reported summary.

This is an estimand choice, not just a calculation choice. Equal weights give each selected update equal influence. Weights proportional to the number eligible give more influence to updates with larger risk sets. Neither weighting is universally correct: each answers a different question.

Two Common Aggregation Targets

For an additive metric such as mean squared prediction error, let \(L_{ik}\) be the loss for subject \(i\) at update \(k\), and let \(\hat M_k\) be its mean among eligible subjects. A general weighted summary is

\hat M_{\mathrm{agg}}=\sum_{k=1}^{K} a_k\hat M_k, \qquad a_k\geq 0,\qquad \sum_{k=1}^{K}a_k=1.

With \(a_k=1/K\), the result is the equal-update average. It describes average performance across the selected updates, giving each update equal weight regardless of how many subjects are eligible. If updates are weighted by \(n_k/\sum_k n_k\), then

\hat M_{\mathrm{row}} =\sum_{k=1}^{K}\frac{n_k}{\sum_{j=1}^{K}n_j}\hat M_k =\frac{\sum_{k=1}^{K}\sum_{i\in E_k}L_{ik}}{\sum_{k=1}^{K}n_k}.

This second quantity is the mean over all eligible subject-update records. It gives more weight to updates with more eligible records, and subjects who contribute at multiple updates can contribute multiple times. It is not a mean over distinct subjects.

Worked Example: Equal-Update and Pooled Brier Scores

Suppose three updates have eligible-set sizes \(100\), \(40\), and \(20\), and their estimated Brier scores are \(0.12\), \(0.20\), and \(0.16\), respectively. Assume the scores use the same outcome definition and horizon, so their weighted combination has a consistent interpretation as a summary across updates.

For the equal-update average, each update receives weight \(1/3\):

\hat M_{\mathrm{equal}} =\frac{0.12+0.20+0.16}{3} =\frac{0.48}{3} =0.160.

For the pooled subject-update average, the total number of records is \(100+40+20=160\):

\hat M_{\mathrm{row}} =\frac{100(0.12)+40(0.20)+20(0.16)}{160} =\frac{12+8+3.2}{160} =0.145.

The equal-update score is \(0.160\); the pooled score is \(0.145\). The pooled result is lower because the largest update, with a score of \(0.12\), receives the most weight. Since lower Brier scores indicate smaller squared prediction errors, the pooled value describes lower average error over eligible subject-update records, not better performance for a typical distinct subject.

The arithmetic relationship above follows by writing each update mean as \(\hat M_k=(1/n_k)\sum_{i\in E_k}L_{ik}\). Multiplying by \(n_k/\sum_j n_j\) cancels \(n_k\) in each term, leaving the total loss divided by the total number of records. This identity applies to additive averages; it does not justify pooling every metric in the same way.

For example, an update-specific AUC compares pairs of subjects within that update’s evaluation population. A weighted mean of update-specific AUCs is a well-defined summary if its update weights are stated. Combining all records first and calculating a single AUC can instead compare subjects from different updates, changing the comparison target. For non-additive metrics, state exactly how the aggregate is calculated and what comparisons it represents.

Changing Horizons and Eligible Populations

A metric at horizon \(h\) concerns a different prediction question from a metric at horizon \(2h\). Likewise, an update-specific metric among subjects event-free at a late update need not describe the population eligible at an early update. Weighting can summarize these results, but it cannot make distinct targets identical. If horizons differ, report them separately unless a composite across horizons is substantively intended and its weights are specified in advance or otherwise justified.

Worked Example: A Composite Across Two Horizons

Suppose a validation analysis reports a Brier score of \(0.08\) at 30 days among 50 eligible subjects and \(0.18\) at 90 days among 20 eligible subjects. The outcome definitions differ because the horizons differ. For illustration, suppose the stated target is a composite that assigns weight \(0.60\) to the 30-day update and \(0.40\) to the 90-day update.

The specified composite is

\hat M_{\mathrm{target}} =0.60(0.08)+0.40(0.18) =0.048+0.072 =0.120.

This value summarizes the two horizon-specific scores using the chosen target weights. By contrast, weighting by the numbers eligible gives

\hat M_{\mathrm{row}} =\frac{50(0.08)+20(0.18)}{50+20} =\frac{4+3.6}{70} \approx 0.1086.

That calculation gives the 30-day result greater weight because its eligible set is larger. It is a possible record-weighted composite, but it is not interchangeable with the stated \(60\%/40\%\) target. Neither number alone establishes that predictions are better at 30 days than at 90 days: horizon and eligible population both differ.

At a fixed horizon, subjects censored before that horizon do not have a fully observed event status. A metric that requires that status must use an appropriate censoring-adjusted estimator rather than treating censored subjects as event-free or silently dropping them. For example, inverse-probability-of-censoring weighting requires an adequately specified censoring survival model, conditional independent censoring given the variables used by that model, and positivity: subjects in the target population must have a nonzero probability of remaining observed as needed. The metric and its censoring adjustment should be defined within each update’s target before aggregating the estimates.

Dependence and Uncertainty

Predictions from the same subject at different updates are generally dependent: they share the subject’s event history and often much of the same longitudinal information. Treating all subject-update records as independent can therefore understate uncertainty. If \(L_{ik}\) denotes a loss, the variance of a sum over records includes covariance terms for repeated records from the same subject:

\operatorname{Var}\!\left(\sum_i\sum_k L_{ik}\right) =\sum_i\sum_k\sum_{\ell}\operatorname{Cov}(L_{ik},L_{i\ell}) +\text{terms for dependence across distinct subjects}.

Even when subjects are independent of one another, the terms \(\operatorname{Cov}(L_{ik},L_{i\ell})\) for updates \(k\) and \(\ell\) of the same subject need not be zero. A standard way to preserve this dependence in uncertainty estimation is to resample subjects as clusters, carrying along all of each selected subject’s eligible updates, predictions, and outcomes. Recalculate the entire aggregate in every bootstrap replicate. Resampling individual subject-update rows independently breaks the observed within-subject structure.

Worked Example: Uncertainty for a Paired Performance Difference

Suppose two prediction procedures are evaluated at the same two updates for four subjects. For each subject, first average the two update-specific losses for each procedure, then calculate procedure A’s average loss minus procedure B’s average loss. Assume the resulting subject-level differences are \(0.02\), \(0.01\), \(-0.01\), and \(-0.02\). Positive values favor procedure B if lower loss is preferred.

The mean paired difference is

\bar d=\frac{0.02+0.01-0.01-0.02}{4}=0.

The sample standard deviation of the four differences is

s_d =\sqrt{\frac{(0.02)^2+(0.01)^2+(-0.01)^2+(-0.02)^2}{4-1}} =\sqrt{\frac{0.001}{3}} \approx 0.01826.

The standard error of the mean difference, using subjects as the independent units, is \(s_d/\sqrt{4}=0.009129\). This small illustration shows how repeated updates can first be kept together within each subject, then compared using subject-level contributions. In a larger validation, a subject-cluster bootstrap can accommodate more complex weighting and metric calculations. The estimate of zero here indicates no difference in the mean paired loss in these four hypothetical subjects; it does not prove equal performance.

Diagnostics, Alternatives, and Limitations

An aggregate can conceal instability. Report update-specific estimates alongside the aggregate, with eligible-set sizes, horizons, and uncertainty where possible. Look for abrupt changes in the eligible population, unusually large weights, sparse event counts, and updates whose results dominate the summary. Also check whether a change in the aggregate is caused by a change in performance or simply by a change in update weights.

Summary	What receives equal or greater weight?	Useful interpretation
Equal-update average	Each selected update receives equal weight	Average performance across a defined set of update times
Pooled subject-update average	Each eligible record receives equal weight	Average additive loss over all eligible records
Target-weighted average	Updates receive weights chosen for a stated target	Performance under a specified update distribution
Separate update-specific results	No cross-update averaging	Performance at each time and horizon, without hiding variation

A subject-weighted summary is another possible target: average each subject’s loss over that subject’s eligible updates, then average those subject-level means across subjects. It gives each subject equal final weight, but the updates contributing to a subject’s mean may differ. This target is not generally equal to the pooled subject-update average, particularly when subjects contribute different numbers of updates. State the chosen target rather than describing any of these summaries simply as “overall performance.”

Changing horizons: a combined score may mix distinct outcome questions. Keep horizon-specific results visible and justify any composite weights.
Changing eligibility: the population at a late update may be a selected subset of the earlier population. Do not interpret differences as temporal improvement alone.
Repeated records: row-based standard errors can be too small when they ignore within-subject dependence. Use subject-level inference or a method that accounts for clustering.
Non-additive measures: do not assume a weighted mean of update-specific metrics is identical to a metric computed after pooling records. Define the calculation and its comparison set.
Limited support: few eligible subjects, few events, or weak follow-up at an update can make its metric unstable; aggregation does not remove that limitation.

The introductory material on temporal validation establishes why predictions must be assessed at the times and horizons at which they are intended to be used. Aggregation adds a further decision: which of those evaluation targets should dominate a summary? A transparent report gives the update-specific results, the eligibility and horizon at each update, the aggregation weights, and an uncertainty method that respects repeated observations within subjects.

Key takeaway: Aggregate only after defining the target represented by the weights. Keep changing populations and horizons visible, and treat subjects—not repeated prediction records—as the independent units when estimating uncertainty unless a method explicitly models their dependence.

Check Your Understanding

Answer each question using the target and weighting assumptions stated.

Two updates have Brier scores \(0.10\) and \(0.22\), with 80 and 20 eligible subjects. Calculate the equal-update and pooled subject-update averages.
Why can the pooled subject-update average give one subject more influence than another?
At a 60-day horizon, a subject is censored at day 25 without an event. Can that subject be coded as event-free through day 60 when calculating an ordinary Brier score? What is needed instead?
Two update-specific AUCs are averaged using stated weights. Why might calculating one AUC after pooling all update records answer a different question?

Solutions. (1) The equal-update average is \((0.10+0.22)/2=0.160\). The pooled average is \([80(0.10)+20(0.22)]/100=(8+4.4)/100=0.124\). (2) A subject eligible at several updates contributes several subject-update records, while a subject eligible at only one contributes one; the pooled record average weights records, not distinct subjects. (3) No. The event status through day 60 is unknown after censoring at day 25, so use an appropriate censoring-adjusted estimator under its assumptions, or another valid method for the stated target. (4) Each update-specific AUC compares subjects within its update-specific population. A pooled AUC may additionally compare records from different updates, so its pairwise comparison target can differ from the weighted average of within-update AUCs.