Why Diagnose Event Projections?
An event projection is a forecast, not a guarantee. Its value depends on whether the assumptions used to predict the number and timing of events remain reasonable as a trial proceeds. Comparing forecasts with observed event counts can reveal a projection that repeatedly predicts too many or too few events. That matters operationally: systematic error can mislead planning for an event-triggered look, even when any single forecast seems plausible.
The purpose of this tutorial is diagnosis: to describe discrepancies and patterns that signal a projection may be miscalibrated. A residual pattern can suggest what to investigate, but it does not, by itself, identify the cause or establish a replacement event-rate assumption.
The comparison is meaningful only when \(O\) and \(E\) refer to the same population, endpoint, observation window, and event-counting rules. In particular, the forecast should be recorded as it stood before the observed count was available. If the projection was updated using the very observations being assessed, the comparison is not an independent check of its forecast performance.
Residuals and Their Scale
A raw error of three events can be important when the forecast standard deviation is two, but unsurprising when it is ten. When the forecast includes a predictive variance, a standardized residual puts the discrepancy on a scale that accounts for the forecast’s expected variability.
For example, suppose a forecast treats the event indicators for \(n\) individuals as independent conditional on their predicted event probabilities \(q_i\). Each indicator \(X_i\) is one if that individual has an event in the specified window and zero otherwise, with \(P(X_i=1)=q_i\). Then \(O=\sum_i X_i\), \(E=\sum_i q_i\), and the predictive variance is \(\sum_i q_i(1-q_i)\). This calculation is conditional on the probabilities and independence assumptions; it does not automatically include uncertainty in estimated event rates or dependencies between individuals.
Why the Standardization Formula Works
The calculation follows from the mean and variance of a sum. For a Bernoulli event indicator, \(E(X_i)=q_i\) and \(\operatorname{Var}(X_i)=q_i(1-q_i)\). If the indicators are independent, the expected count is the sum of their expectations and the count variance is the sum of their variances:
Consequently, dividing \(O-E\) by the square root of this variance expresses the observed discrepancy in predictive standard-deviation units. If the count distribution is approximately normal, a standardized residual can be compared informally with a standard normal scale. This is a diagnostic approximation, not an automatic formal test: small expected counts, estimated probabilities, model uncertainty, or dependence can make a normal reference misleading.
Worked Example: A Single Forecast That Underpredicts
Suppose a hypothetical trial forecast predicts 40 events by a specified cutoff, with predictive variance 36 events squared. The observed count at that cutoff is 48. Assume the forecast and observed count use the same cutoff and event definition, and that the stated variance is appropriate for the predictive distribution.
The signed error is \(R=O-E=48-40=8\) events. Since the error is positive, the forecast underpredicted the count.
The predictive standard deviation is \(SD_{\mathrm{pred}}=\sqrt{36}=6\) events. Therefore, \(Z=(48-40)/6=1.33\). The observed count is 1.33 predictive standard deviations above the forecast mean. This is a discrepancy worth recording, but one result alone does not establish systematic underprediction; ordinary forecast variation can produce a positive error.
Look for Patterns Across Time
A sequence of forecast errors is often more informative than one difference. Persistent positive residuals indicate that observed counts have tended to exceed forecasts; persistent negative residuals indicate that forecasts have tended to exceed observed counts. A residual plot can show direction, changes over time, and isolated departures. Plot the residual against the cutoff date or the period being forecast, and include a horizontal reference line at zero. If useful, also plot standardized residuals, while stating how their predictive standard deviations were obtained.
For diagnosing changing accrual over successive time windows, disjoint-period counts are often easier to interpret than cumulative counts. Let \(O_j\) and \(E_j\) denote observed and predicted events in period \(j\), and define \(R_j=O_j-E_j\). Each period should have a clear start and end, and events should be counted only once. A run of residuals with the same sign can signal a systematic discrepancy, but the size and uncertainty of those errors still matter.
Worked Example: A Run of Overpredictions
A hypothetical team records forecasts for three successive, non-overlapping periods. The predicted counts are 20, 25, and 30 events; observed counts are 16, 21, and 26. The corresponding predictive standard deviations are 4, 5, and 6 events. Assume the periods and event definitions match, and treat these standard deviations as appropriate for each period’s predictive count.
The signed errors are \(16-20=-4\), \(21-25=-4\), and \(26-30=-4\) events. Each is negative, so the forecast overpredicted in all three periods.
The standardized residuals are \(-4/4=-1.00\), \(-4/5=-0.80\), and \(-4/6=-0.67\). None is large on its own under a roughly normal predictive model. However, all three point in the same direction. A residual plot would display a run below zero. This pattern is a reason to investigate systematic overprediction, not proof of its cause. For example, confirm that the forecast windows and adjudicated event counts were aligned before attributing the pattern to event-rate assumptions.
Cumulative counts require special care. The observed cumulative total at a later cutoff includes events counted at earlier cutoffs. Thus residuals at successive cumulative cutoffs are generally dependent: they share much of the same event history. Treating each cumulative residual as an independent observation exaggerates the amount of evidence in a sequence. To inspect changes between cutoffs, calculate period increments by subtracting successive cumulative totals and compare those increments with the corresponding forecasts.
Worked Example: Turning Cumulative Counts into Period Checks
A hypothetical forecast gives cumulative event counts of 30, 55, and 80 at three cutoffs. The observed cumulative counts are 32, 59, and 87. The forecast and observed totals refer to the same trial and event definition at each cutoff.
The cumulative signed errors are \(32-30=2\), \(59-55=4\), and \(87-80=7\) events. They are all positive, but they are not three independent discrepancies: each later total contains the earlier events.
Instead, compare increments. Forecast increments are \(30\), \(55-30=25\), and \(80-55=25\) events. Observed increments are \(32\), \(59-32=27\), and \(87-59=28\) events. The period errors are therefore \(32-30=2\), \(27-25=2\), and \(28-25=3\) events. These increments show that the later positive cumulative error reflects positive discrepancies in each of the three periods. A complete uncertainty assessment would use appropriate predictive variances for those periods; the cumulative totals alone do not supply them.
Diagnostics, Alternatives, and Limitations
A residual plot is a diagnostic display, not a substitute for checking the data behind it. Before interpreting a run or a large residual, verify that the forecast was archived before the cutoff; the cutoff dates and operational lag match; event adjudication and data-cleaning rules have not changed; and the observed and predicted counts use the same eligibility, censoring, and follow-up conventions. A mismatch in any of these can create an apparent projection error even if the underlying event process has not changed.
Compare the main diagnostic approaches according to the question they answer:
| Approach | What it helps show | Main caution |
|---|---|---|
| Raw signed residuals | Direction and size of forecast errors in event units | Does not account for how variable the forecast count is |
| Standardized residuals | Error size relative to predictive variability | Depends on a suitable variance model and may omit model uncertainty |
| Residual plot over time | Runs, shifts, and isolated discrepancies | Repeated cumulative residuals are dependent |
| Observed and predicted counts side by side | Operational scale and overall trajectory | Agreement in totals can hide offsetting period errors |
A plot of observed and predicted counts is useful for seeing the overall trajectory, but visually close cumulative curves can conceal an early overprediction followed by a later underprediction. Period residuals help expose such changes. Conversely, individual period residuals may be noisy, so a single period’s sign should not be treated as a definitive diagnosis. A predictive interval can provide another comparison: an observation outside an appropriate interval is more surprising under that predictive model than one inside it, but an interval crossing the observation does not prove the model is calibrated.
Residual diagnostics have limits. They can reveal that forecasts repeatedly miss in a direction, but they cannot determine whether the reason is a changed event rate, enrollment timing, loss to follow-up, delayed adjudication, or an incorrect variance assumption. Counts also discard individual timing information: different event-time patterns may produce the same count at a cutoff. Finally, a standardized residual near zero does not validate every feature of the model; errors in different periods can cancel in an aggregate.
Common Mistake
Wrong reasoning: “The cumulative residuals were positive at three cutoffs, so there are three independent pieces of evidence that the projection underpredicts.” This ignores that each later cumulative count includes earlier events. Correction: acknowledge the dependence, examine the forecast errors in non-overlapping periods when the question concerns changes over time, and use an uncertainty calculation appropriate to those periods. Even a run of same-sign period errors is a diagnostic signal, not a causal explanation.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A forecast predicts 52 events and 46 are observed. What is the signed projection error, and what direction does it indicate?
- A forecast predicts 25 events with predictive standard deviation 5, and 35 are observed. Calculate the standardized residual and interpret it in context.
- Why should successive cumulative residuals not normally be treated as independent observations?
- Observed counts exceed forecasts in four successive non-overlapping periods. What does this pattern suggest, and what does it not establish?
- Give two checks to make before interpreting a large residual as evidence of a projection problem.
Solutions. (1) \(R=46-52=-6\) events, so the forecast overpredicted by 6 events. (2) \(Z=(35-25)/5=2.00\). The observed count is two predictive standard deviations above the forecast mean; the positive sign indicates underprediction, subject to the predictive model’s assumptions. (3) Later cumulative counts include earlier events, so successive errors share data and are dependent. (4) It suggests persistent underprediction across those periods and warrants investigation; it does not prove the cause or rule out a chance run. (5) For example, check that the forecast was recorded before the data cutoff and that observed and predicted counts use the same cutoff, event definition, adjudication status, and follow-up rules.