Evaluating Incremental Prediction Beyond Clinical Covariates

Why incremental prediction needs a fair comparison

A polygenic risk score (PRS) may be associated with an outcome and still add little useful predictive information once age, family history, and other clinical covariates are known. The relevant question is therefore comparative: how much do predictions improve when the PRS is added to the clinical predictors? Answering it requires more than reporting a performance number for the expanded model. The clinical-only and clinical-plus-PRS models must be evaluated on the same people, with the same outcome definition and prediction horizon.

This comparison concerns prediction, not causation. A change in model performance does not show that the PRS causes the outcome, nor does it by itself establish that using the score improves decisions. It describes how well the two sets of predictions perform in a specified evaluation population. The introductory material on discrimination metrics for binary outcomes and calibration diagnostics for polygenic scores provides the definitions of those concepts; here the emphasis is on comparing them fairly.

Define the models and evaluation target

Let \(Y_i\) indicate whether person \(i\) experiences the binary outcome during a prespecified time horizon. Let \(X_i\) denote the clinical covariates and \(S_i\) the PRS. A common pair of logistic prediction models is:

Here \(M_0\) is the clinical-only model and \(M_1\) adds the PRS. The coefficients are estimated in training data, and each fitted model produces a predicted probability, \(\hat p_{0i}\) or \(\hat p_{1i}\), for each person in the evaluation data. Any model selection, score tuning, or choice of transformations must be completed without using evaluation outcomes. If the PRS or its tuning choices were selected using those outcomes, the evaluation can be optimistically biased.

The comparison assumes that both predictions refer to the same outcome definition and time horizon, and that both models are assessed on the same evaluation people. For the results to describe a target population, the evaluation people must represent that population or be appropriately weighted. Independent evaluation is especially important: a model’s performance on the observations used to fit or tune it generally overstates its performance on new observations.

Compare discrimination on the same cases and controls

For a binary outcome, the area under the receiver operating characteristic curve (AUC) measures ranking discrimination. It is the probability that a randomly selected case receives a higher predicted risk than a randomly selected noncase, with ties counted as one-half. AUC does not assess whether predicted probabilities are numerically accurate. Adding a PRS can change rankings and therefore AUC, but a useful prediction comparison also asks whether predicted risks are calibrated and whether overall prediction error falls.

Because both models predict the same evaluation people, their AUCs are paired. Report both AUCs and their difference, with uncertainty when relevant. A paired DeLong procedure or a bootstrap that resamples evaluation people and recomputes both AUCs can quantify uncertainty in the difference. Treating the two AUCs as if they came from unrelated samples ignores their shared cases and controls. A small change may also be estimated imprecisely, particularly when the evaluation set has few cases.

Assess calibration and overall prediction error

Calibration concerns agreement between predicted probabilities and observed outcome frequencies. For a calibration regression, let \(\eta_i=\operatorname{logit}(\hat p_i)\) be a model’s predicted log-odds. The evaluation data can be used to estimate the diagnostic relationship \(\operatorname{logit}\{P(Y_i=1)\}=a+b\eta_i\). Ideal calibration has intercept \(a=0\) and slope \(b=1\). The intercept indicates systematic under- or overprediction when the slope is fixed at one; the slope assesses whether predictions are too extreme or not extreme enough. These are diagnostics on the evaluation predictions, not instructions to refit the prediction model on the evaluation outcomes.

The Brier score and logarithmic loss summarize probability prediction error, rewarding accurate probabilities and penalizing inaccurate ones. For \(n\) evaluation people, the Brier score is the mean squared error of the predicted probabilities. Log loss assigns a larger penalty to confident predictions that are wrong. Both are lower-is-better metrics. Unlike AUC, they reflect probability accuracy as well as ranking, and both depend on the outcome prevalence in the evaluation population.

A reference-based summary can put these errors in context. For example, the Brier skill score is \(1-\mathrm{Brier}_{\text{model}}/\mathrm{Brier}_{\text{reference}}\); a positive value means the model has lower Brier error than the chosen reference. A log-loss-based McFadden-style pseudo-\(R^2\) is \(1-\mathrm{LogLoss}_{\text{model}}/\mathrm{LogLoss}_{\text{null}}\). These are relative reductions in specified prediction losses, not proportions of outcome variance explained in the ordinary linear-regression sense. State the reference prediction explicitly. For an external evaluation, a constant reference probability can be based on the training prevalence; estimating it from evaluation outcomes would use information unavailable when making predictions.

A short justification for paired comparisons

For a per-person loss, the incremental result is naturally computed person by person. Define \(d_i=\ell(\hat p_{0i},Y_i)-\ell(\hat p_{1i},Y_i)\), where positive values favor the expanded model for a lower-is-better loss. The estimated mean improvement is \(\bar d=n^{-1}\sum_i d_i\). Since both predictions are evaluated against the same \(Y_i\), their errors are generally related. The variance of a difference includes this dependence:

This is why uncertainty should be estimated from paired predictions, for example by resampling people and retaining both predictions and the outcome for each resampled person. Separate resampling of the two models would discard the covariance term and would no longer represent the comparison actually made. For AUC, the paired procedure must also preserve the shared case and control sets.

Diagnostics, alternatives, and limitations

A useful report shows the two model performances side by side, the direction and size of each change, and uncertainty for the paired change. AUC can improve while calibration or log loss worsens; this is not a contradiction, because the metrics assess different properties. Conversely, a small AUC change can accompany a reduction in probability error. No single metric is a complete description of predictive performance.

Check that the same people, outcome coding, follow-up horizon, and missing-data rules were used for both models. Different evaluation samples can confound a model comparison with differences in case mix or outcome prevalence. In a case-control evaluation, the sampled prevalence generally does not represent the target prevalence: AUC can still describe ranking in the sampled cases and controls, but calibration, Brier score, and log loss require an appropriate sampling design or weighting to represent target-population risks. Report the evaluation prevalence and any weights used.

A nested-model likelihood-ratio test asks whether adding the PRS improves fit under a specified model and its assumptions; it is not a substitute for held-out predictive evaluation. Likewise, coefficient significance for \(\delta\) does not quantify the change in calibration or prediction error. AUC is useful when ranking is the focus, while calibration plots and calibration parameters address probability accuracy, and Brier score or log loss gives an overall proper scoring rule. The suitable comparison depends on the intended prediction task.

A common mistake is to compare the clinical model’s AUC in one sample with the expanded model’s AUC in another and attribute the entire difference to the PRS. The samples may differ in prevalence, clinical spectrum, or measurement quality. The correction is to generate both models’ predictions for the same held-out people and compare the paired metric differences there. Another mistake is to call a positive pseudo-\(R^2\) the proportion of outcome variance explained; it is a relative change in a defined likelihood-based loss, not an ordinary variance decomposition.