Incremental Accuracy Beyond Clinical Predictors

Why Compare Prediction Accuracy?

A clinical risk model assigns a predicted probability to each person. Adding a polygenic score may change those predictions, but a change alone does not show whether they became more accurate. Accuracy scores provide a way to compare predicted probabilities with outcomes that are later observed. This tutorial focuses on two such scores for binary outcomes: the Brier score and the negative logarithmic score.

The comparison is between a clinical-only model and a polygenic-plus-clinical model evaluated for the same outcome, time horizon, and people. A lower average score indicates better performance under that scoring rule in that validation sample. It does not, by itself, show that adding genetic information improves decisions, that genetic effects are causal, or that the predictions will perform equally well in a different population.

Predictions, Outcomes, and Scoring Rules

Suppose \(Y_i\) indicates whether person \(i\) experiences the event by a prespecified time horizon: \(Y_i=1\) if the event occurs and \(Y_i=0\) otherwise. Let \(\hat p_i^C\) be the event probability predicted by the clinical-only model, and let \(\hat p_i^{C+G}\) be the prediction from the model combining clinical predictors and a polygenic score. These are predicted probabilities, not observed outcomes.

For an event, the negative logarithmic score reduces to \(-\ln(\hat p_i)\); for a non-event, it reduces to \(-\ln(1-\hat p_i)\). The logarithm here is the natural logarithm, so the score is expressed in nats. A prediction of zero or one can give an infinite score if it assigns probability zero to the outcome that occurs. In practice, the log score therefore penalizes confidently wrong predictions especially heavily.

The Brier score is bounded between zero and one. It is zero for a perfect probability prediction of the observed binary outcome and approaches one for a maximally wrong prediction. It is measured on the squared probability scale, not in units such as years or cases. The negative logarithmic score is zero when the assigned probability for the observed outcome is one, and grows without bound as that probability approaches zero.

Conditions for a Fair Comparison

Before interpreting either score, define the prediction target precisely. For example, the target might be whether a specified disease occurs within five years. Both models must predict that same event over that same horizon, and their scores must be calculated on the same validation individuals. If one model is evaluated on a different sample or a different outcome definition, the score difference combines model performance with differences in the evaluation data or target.

The models should be fit without using the outcomes in the validation sample. Their predictions must be available as probabilities for each evaluated person. For the basic formulas here, assume each person’s binary outcome is ascertained through the full time horizon, with no unresolved censoring or missing outcome. If some people leave follow-up before the horizon or their outcome is unknown, simply treating them as non-events can bias the comparison; methods that account for incomplete follow-up are then needed.

The comparison is paired: each person contributes one score under each model. This holds the validation sample fixed, so differences reflect the predictions rather than a change in which people were evaluated. The models should also be developed and applied consistently, including any fitting or recalibration needed to produce their probabilities. A fair comparison does not require the models to have the same number of predictors, but it does require the same prediction target and an evaluation process that does not leak validation outcomes into model development.

Why Lower Scores Indicate Better Probability Predictions

A scoring rule is called proper when, on average, it rewards reporting the true event probability. This property explains why these scores evaluate probabilities rather than merely whether a model assigned the larger prediction to people who experienced the event. Let \(q\) denote the true event probability for a person with given information, and let \(p\) be a proposed forecast. The Brier score’s expected value over the binary outcome is:

The first term, \(q(1-q)\), does not depend on the forecast. The second term, \((p-q)^2\), is nonnegative and is minimized when \(p=q\). Thus, for the Brier score, the true event probability minimizes expected loss.

For the negative logarithmic score, the expected value is:

The last two terms together are the Kullback–Leibler divergence from a Bernoulli distribution with probability \(q\) to one with probability \(p\). This divergence is nonnegative and is zero when \(p=q\). Therefore the expected negative log score is also minimized by the true probability. At \(q=0\) or \(q=1\), the same conclusion follows by taking the appropriate limiting values.

This justification concerns expected scores when forecasts are assessed over the outcomes they predict. In a finite validation sample, random variation can make a model with less accurate underlying probabilities achieve a lower observed mean score. A score difference is an estimate of comparative performance, not a guarantee about the population-level ordering.

Reading an Accuracy Improvement Carefully

The mean score difference for paired predictions can be written as the average of the person-level differences. For the Brier score, one useful convention is to define the improvement as the clinical-only score minus the combined-model score. A positive difference then favors the combined model. The same convention can be used for the negative log score.

The calculation should use the same individuals in both terms. A positive \(\Delta_{\text{Brier}}\) means the combined model had a lower mean Brier score in the evaluated sample; a negative value means its mean score was higher. The difference is not a percent improvement unless it is explicitly divided by a defined reference score and reported as such.

An average score can conceal variation across people. Some predictions may improve while others worsen, and a few extreme errors can strongly affect the log score. It is therefore useful to inspect the paired individual differences and, where relevant, results across prespecified groups. To quantify uncertainty in an average difference, one can use a paired bootstrap: resample validation individuals, retaining both models’ predictions and the outcome together for each person, then recalculate the difference in each resample. The resulting interval describes sampling uncertainty under the resampling design; it does not correct biased sampling, outcome misclassification, or a mismatch between the validation and intended-use populations.

What Accuracy Scores Do Not Establish

Brier and negative log scores summarize different penalties applied to predicted probabilities. Neither score, alone, diagnoses why performance differs. A lower average can reflect better probability calibration, better separation of people with and without events, or both. A lower score in one validation sample also does not establish that the polygenic score itself caused the improvement: model comparisons are predictive evaluations, not causal experiments.

Most importantly, an average accuracy gain is not the same as clinical utility. It does not show whether predictions change a treatment choice, whether that change helps more people than it harms, or whether the model is suitable for a particular clinical setting. Such conclusions require evidence about decisions and their consequences, as well as validation in the intended population. Accuracy scores answer a narrower question: how well did each set of predicted probabilities match the observed binary outcomes under the chosen scoring rule?

A common mistake is to say, “The combined model has a lower score, so adding the polygenic score improves clinical care.” The first part may be supported by a careful validation comparison; the second does not follow from it. A more precise conclusion is: “In this validation sample and for this outcome horizon, the polygenic-plus-clinical model had a lower mean Brier score than the clinical-only model. This comparison alone does not establish better clinical decisions or benefit.”