Why Validate Risk Predictions With a Brier Score?
A competing-risk model produces a probability, such as the probability of experiencing cause 1 by 12 months, in the presence of other possible event types. A validation measure should assess these probabilities against what is subsequently observed. The Brier score does this by averaging squared differences between predicted probabilities and binary outcomes. Lower values indicate better average probabilistic accuracy, but only when the outcome and handling of censoring are defined appropriately.
At a fixed horizon, the score evaluates one cause-specific prediction. It does not by itself say whether a model ranks people well at every point in follow-up, whether any particular subgroup is well predicted, or why predictions are inaccurate. Its value is a summary of prediction error for a specified cause, population, and time.
A competing event before \(t\) is an observed outcome, not censoring: it establishes that the person did not experience cause \(k\) by that horizon, so \(Y_k(t)=0\). A person who remains event-free past \(t\) also has outcome zero. By contrast, a person censored before \(t\) has an unknown outcome at \(t\); treating that person as a known zero generally biases the score.
Inverse-Probability-of-Censoring Weighting
With right censoring, the inverse-probability-of-censoring weighted (IPCW) Brier score uses only outcomes that are known, weighted to represent people with similar follow-up who were censored. Write \(\tilde T_i\) for the observed time, \(\Delta_i=1\) if an event was observed and 0 if censored, and \(G(s\mid Z_i)\) for the probability of remaining uncensored through time \(s\), conditional on censoring-model covariates \(Z_i\). For a cause-\(k\) prediction \(\hat F_{ik}(t)\), an estimator is
An observed event by \(t\), of any cause, contributes the squared error for its known binary outcome. An individual observed beyond \(t\) contributes the squared error for outcome zero. Censoring before \(t\) contributes no direct term. The inverse weights compensate for the chance that an otherwise similar individual would have been censored before the relevant time. In practice, the censoring survival probabilities are estimated, and a package may implement details of the estimator and its uncertainty calculation.
This approach requires censoring to be independent of the event outcome, conditional on the covariates used in the censoring model. It also requires positive probabilities of remaining uncensored: the relevant \(G\) values must not be zero, and values close to zero can make weights very large. Predictions must refer to the same cause and horizon as the observed outcomes. Validation data should represent the population in which performance is intended to be assessed; evaluating on the model-fitting data can give an overly optimistic assessment.
Worked Example: Calculating an IPCW Score at 12 Months
Consider six hypothetical validation subjects and cause 1. Their predicted 12-month risks are, respectively, 0.40, 0.20, 0.60, 0.10, 0.50, and 0.30. Subject 1 has a cause 1 event before 12 months with \(\hat G(\tilde T_1-)=0.80\). Subject 2 has a competing event with \(\hat G(\tilde T_2-)=0.90\). Subject 3 has a cause 1 event with \(\hat G(\tilde T_3-)=0.75\). Subject 4 is observed event-free beyond 12 months and \(\hat G(12)=0.80\). Subject 5 is censored before 12 months. Subject 6 has a competing event before 12 months with \(\hat G(\tilde T_6-)=0.80\).
Subjects 1 and 3 have outcome 1; subjects 2, 4, and 6 have outcome 0. Subject 5's outcome at 12 months is unknown, so that subject has no direct contribution. The contributions for the others are:
The score divides the sum by the original sample size \(n=6\), including the censored subject:
The competing events count as known zeros for the cause 1 outcome. Omitting them as though they were censored would discard observed information and would change the estimand.
Why Squared Probability Error Is Informative
The Brier score is a proper scoring rule: in expectation, it is minimized by reporting the true event probability. To see this, let \(Y\) be a binary outcome with conditional probability \(q=P(Y=1\mid X)\), and let a model report \(p\). The conditional expected squared error is
The first term does not depend on \(p\), and the second is nonnegative and equals zero only when \(p=q\). Thus systematically inaccurate probabilities increase expected loss. Under censoring, IPCW aims to estimate this same prediction loss despite some outcomes being unobserved, provided the censoring assumptions hold.
Worked Example: Comparing Two Predictions When Outcomes Are Known
Suppose four hypothetical subjects are all followed through one year. Their cause 1 outcomes are \(1,0,0,1\). Model A predicts risks \(0.70,0.20,0.30,0.60\), while Model B predicts \(0.50,0.40,0.10,0.80\). No IPCW adjustment is needed because all four outcomes at the horizon are known.
For Model A, the squared errors are \(0.09,0.04,0.09,0.16\), so
For Model B, the squared errors are \(0.25,0.16,0.01,0.04\), so
Model A has the lower observed Brier score by \(0.115-0.095=0.020\) on this sample. This is a descriptive comparison, not proof that Model A will perform better in another sample. Both models are assessed on the same individuals, so uncertainty in their difference should account for the paired predictions.
Using riskRegression::Score
The earlier material on predicting competing-risk probabilities with riskRegression established how to obtain cause-specific risk predictions. Here, the task is to evaluate those predictions on validation data. Fit candidate models using training data, then pass the fitted objects and the separate validation data to riskRegression::Score. The outcome formula identifies the time and status variables; cause and times specify the event type and horizons being scored.
For example, assuming the fitted objects are named fit_csc and fit_fg, and the validation data contain time and status, a call can be written as:
sc <- riskRegression::Score(object = list(CSC = fit_csc, FineGray = fit_fg), formula = prodlim::Hist(time, status) ~ 1, data = validation, times = c(365, 730), cause = 1, metrics = "Brier", cens.method = "ipcw", cens.model = "cox")
This example uses days as the time unit, so the requested horizons are 365 and 730 days. Status coding must agree with the fitted models: censoring must be represented as censoring, and the selected cause must be the event code intended for evaluation. The censoring model should be adequate for the validation setting. If censoring depends on measured covariates, a marginal censoring model may not be sufficient; if a conditional model is used, its covariates should capture the relevant dependence.
Inspect the returned Brier-score results, commonly available in the Brier component of the result object, and check that each model, cause, and horizon appears as intended. Request or inspect uncertainty estimates as appropriate for the installed package version. If the package's model-contrast option is used, it compares models using their predictions on the same validation sample; do not infer a difference merely because two separate confidence intervals overlap or do not overlap.
Worked Example: Reading a Two-Horizon Model Comparison
Suppose a hypothetical validation run reports the following Brier scores for cause 1. Both candidate models were fitted on training data and scored on the same validation individuals.
| Horizon | Cause-specific model | Fine-Gray model |
|---|---|---|
| 365 days | 0.142 | 0.151 |
| 730 days | 0.184 | 0.176 |
At 365 days, the cause-specific model has the lower score: \(0.151-0.142=0.009\). At 730 days, the Fine-Gray model has the lower score: \(0.184-0.176=0.008\). Thus these hypothetical results do not identify one model as uniformly better across the two horizons. The scores are dimensionless averages of squared probability errors; they are not percentages of people correctly classified.
A defensible comparison also examines uncertainty in the paired score differences and considers whether the chosen horizons and validation population match the intended use. If the horizon was selected after inspecting many candidate times, the reported comparison may overstate how persuasive the apparent advantage is.
Diagnostics, Alternatives, and Limitations
Before interpreting a score, check that the predictions are finite probabilities between zero and one, that the event coding is correct, and that every requested time is supported by the observed follow-up. Examine the estimated censoring survival probabilities or weights: extremely large weights indicate that a small number of observations may dominate the result. Also compare the censoring model with the study's follow-up process. A poor censoring model can leave residual bias even when the outcome model is well fitted.
The Brier score is an overall measure combining aspects of calibration and discrimination. It is not a substitute for examining calibration, which is the focus of the following material, and it does not isolate the ability to rank individuals. A time-dependent AUC is a common alternative when discrimination is the specific target. Reporting both can be useful because a model may rank subjects well while assigning inaccurate probabilities, or vice versa. Neither measure removes the need for a representative validation design.
A score at one horizon cannot summarize performance at all follow-up times. Scores can also depend on the cause, the mix of competing events, the censoring distribution, and the validation population. Comparisons across populations or horizons should therefore be described in context rather than treated as universal rankings.
Check Your Understanding
Answer each question, then check the solutions below.
- At a fixed horizon, what binary outcome is used for a cause-\(k\) Brier score when a person experiences a different cause first?
- Why does a subject censored before the horizon not simply receive a zero outcome?
- For a known outcome of 1 and predicted risk 0.30, calculate the unweighted squared error.
- What does it mean if one model has a lower Brier score at 365 days but a higher score at 730 days?
- Name two conditions or diagnostics important for interpreting an IPCW score.
Solutions. (1) The outcome is zero: the person did not experience cause \(k\) by the horizon. (2) The person's event status by that horizon is unknown, so treating it as zero falsely labels an unobserved outcome as event-free. (3) The error is \((1-0.30)^2=0.49\). (4) The models' relative performance differs by horizon; neither is uniformly better based on those results. (5) Examples include conditional independence of censoring given the censoring-model covariates, positivity of the probability of remaining uncensored, adequate censoring-model fit, and absence of extreme IPCW weights.