Checking Probability Coherence

Why Check Probability Coherence?

A prediction can look plausible when each number is read separately and still be impossible when the numbers are considered together. For example, a model might report a 70% risk of one event, a 40% risk of a competing event, and a 5% probability of remaining event-free at the same horizon. Those three outcomes cannot describe one probability distribution: their probabilities add to 115%.

A coherence check is therefore a basic audit of absolute-risk predictions. It does not tell us whether predictions match observed outcomes, but it can reveal a computation or reporting error before predictions are interpreted, compared across profiles, or used in decisions.

These statements concern predictions for the same profile, horizon, outcome definition, and target population. The \(F_k\) values are probabilities of distinct first-event outcomes. The event-free probability is the probability of the remaining outcome: no event from any of the included causes by \(t\). Censoring is not an additional event category in this probability partition; it is incomplete observation of the event process.

If the reported causes do not include every event type that can occur, the listed risks need not add to one minus the reported event-free probability. The missing event types must either be included in the sum or represented by an explicit residual probability. Before diagnosing a numerical discrepancy, check that the outcome categories really are exhaustive and that all quantities refer to the same horizon and conditioning information.

Where the Identity Comes From

At a fixed horizon, a person is in exactly one of \(K+1\) states: an event from cause 1 by \(t\), an event from cause 2 by \(t\), and so on, or no included event by \(t\). Under the stated outcome definition, these events are mutually exclusive and collectively exhaustive. Adding their probabilities gives one:

Replacing each probability by its notation gives the coherence identity. The argument is a partition argument, not a modeling assumption that can be relaxed to permit a total above or below one. The modeling and data assumptions matter because they determine whether the reported quantities actually refer to this partition.

A useful computational check comes from successive time intervals. Suppose \(S_{j-1}\) is the probability of having had no event by the start of interval \(j\). Let \(q_{kj}\) be the conditional probability of cause \(k\) during interval \(j\), given no event before that interval, and let \(q_{0j}\) be the conditional probability of no event during the interval. If these conditional probabilities are coherent, then \(\sum_k q_{kj}+q_{0j}=1\). The interval contributes \(S_{j-1}q_{kj}\) to cause \(k\)'s cumulative incidence, and updates event-free probability to \(S_j=S_{j-1}q_{0j}\). Thus:

If the identity held at the start of the interval, it holds at the end. At time zero, the cumulative incidences are zero and event-free probability is one, so the identity holds initially. Repeating the update therefore preserves coherence at every interval. This also explains why cause-specific cumulative incidences are not obtained by simply adding conditional event probabilities across intervals: each interval's conditional probabilities apply only to people still event-free at its start.

Practical Checks on a Prediction Table

For each row of a prediction table, first confirm the covariate profile and time horizon. Then check the range of every reported probability and the total across the complete set of outcomes. The relevant comparison is between \(\sum_k F_k(t\mid x)\) and \(1-S(t\mid x)\), or equivalently between \(\sum_k F_k(t\mid x)+S(t\mid x)\) and one. Use unrounded values for this calculation whenever possible.

A small discrepancy in a table of rounded values may be harmless. For instance, displayed values rounded separately to two decimal places need not sum exactly to 1.00 even if the unrounded predictions do. Determine whether the discrepancy is compatible with the stated rounding precision. In contrast, a large discrepancy, a negative value, or a value above one calls for investigation of the underlying calculation rather than an appeal to rounding.

A useful audit follows the prediction from its inputs to its reported output:

Coherence Across Causes and Standardized Predictions

Predictions can be assembled in different ways. When cause-specific cumulative incidences are calculated together from a single competing-risks process, the event-free probability and cause-specific risks should obey the same partition identity. When separate models or calculation routines produce the reported quantities, coherence should be checked rather than presumed. For example, independently generated risks and survival estimates might use different horizons, event coding, or input covariates.

Standardization averages profile-specific predictions using the same nonnegative weights, with weights summing to one. If each profile's predictions are coherent and the same profiles and weights are used for every cause and the event-free probability, the standardized results remain coherent. Indeed, if the weights are \(w_i\), then:

This result depends on using common weights and a common outcome definition. Averaging each cause over one set of individuals and event-free probability over another set does not have this guarantee. If standardization is involved, check both the profile-level predictions and the weighting procedure.

What Coherence Does Not Establish

A coherent prediction is mathematically possible, not necessarily accurate. Predictions can satisfy every probability identity and still systematically overpredict or underpredict observed risks. Coherence is also distinct from uncertainty: a confidence interval for an estimated risk can have an endpoint outside the probability range when constructed using an unsuitable approximation. That is a warning about the interval procedure or its scale, not evidence that the underlying true probability is negative or greater than one.

Coherence checks should be applied to the point predictions being reported, including any standardized predictions. They do not replace assessment of model fit, calibration, data support, or uncertainty. Nor do they establish a causal interpretation: the check concerns whether the reported numbers can jointly represent probabilities for the stated prediction target.

Common Mistakes and Failure Modes

• Checking only the sum. A total of one does not excuse a negative component, as the first worked example shows. Check every probability and the total.
• Comparing unlike quantities. A 5-year cause-specific risk cannot be reconciled with a 10-year event-free probability. Align the horizon, profile, and outcome definition first.
• Treating a missing cause as if it had zero risk. If causes are not exhaustive, the unlisted outcomes need a residual category; do not force the listed risks and event-free probability to sum to one without that category.
• Using rounded values to diagnose the underlying computation. Small differences can result from rounding separate entries. Recheck full-precision predictions before concluding that the calculation failed.
• Clipping an impossible prediction without review. Changing a negative risk or a value above one by hand can conceal the source of error and disrupt the total. Trace the computation and recompute consistently.
• Calling coherence evidence of good prediction. A set of numbers can sum correctly and still be poorly calibrated or imprecise. Coherence is a necessary internal check, not a complete model evaluation.