Comparing Score Performance Across Outcome Scales

Why One Portability Comparison Needs Several Metrics

A prediction score may perform differently across ancestry-defined target groups, but “performance” is not a single property. For a quantitative outcome, a common summary is the fraction of outcome variation explained by predictions. For a binary outcome, discrimination describes how well predictions rank cases above controls, while calibration asks whether predicted probabilities agree with observed frequencies. These summaries answer different questions, so a difference in one does not by itself establish a difference in another.

The comparison is meaningful only when the outcome is defined consistently. For a quantitative trait, the groups should be evaluated on the same trait definition and measurement units. For a binary outcome, the case definition and prediction horizon should match. The validation samples and metric calculations must also represent the target groups being compared. A score evaluated against different outcomes, thresholds, or sampling schemes does not yield a clean comparison of portability.

What the Metrics Measure

For a quantitative outcome \(Y\), let \(\hat{Y}_{ig}\) be the prediction for person \(i\) in group \(g\), and let \(\bar{Y}_g\) be the observed mean in that group. A common validation measure is predictive \(R^2\), defined as one minus the ratio of the squared prediction errors to the total squared deviations of outcomes from their group mean:

The numerator is the residual sum of squares, and the denominator is the total sum of squares. This predictive \(R^2\) compares a model’s prediction errors with the variation around the group mean. It can be negative on validation data: this occurs when the squared prediction errors exceed the total sum of squares, so the predictions perform worse by this criterion than using the group mean as a constant prediction. Unlike the squared correlation, predictive \(R^2\) depends on prediction errors as well as the association between predictions and outcomes.

For a binary outcome, write \(Y=1\) for a case and \(Y=0\) for a control, and let \(\hat{p}\) be the predicted probability of being a case. The area under the receiver operating characteristic curve (AUC) measures discrimination: it is the probability that a randomly selected case receives a higher prediction than a randomly selected control, with a tie counted as one-half. The Brier score is the average squared error of the predicted probabilities. Lower Brier scores indicate smaller average squared probability errors.

Calibration concerns agreement between predicted probabilities and observed outcomes. For example, calibration can be summarized by fitting a logistic calibration model in which the outcome is predicted from the logit of the original predicted probability. Ideal calibration has intercept zero and slope one. These summaries are not ranking summaries: predictions can rank cases and controls well while being systematically too high or too low.

Worked Example: Equal Absolute Errors, Different Explained Variance

Discrimination Is Not Calibration

AUC depends on the ordering of predictions across cases and controls. Any strictly increasing transformation of the predictions preserves that ordering and therefore preserves AUC, although it can change the numerical probabilities substantially. Calibration, in contrast, is about whether those probabilities correspond to observed risks. The Brier score also uses the probability values themselves, not just their order.

Prevalence and the Brier Score

AUC summarizes comparisons between cases and controls, rather than the overall proportion of cases. The Brier score averages errors across all individuals, so the proportions of cases and controls affect how much each class contributes. As a result, a Brier-score difference between target groups can reflect different prevalence as well as differences in prediction quality. This is particularly important when a validation sample does not represent the target population’s case prevalence. Evaluation should use a design or weighting appropriate to the intended target population; metrics from differently ascertained samples should not be interpreted as though they automatically describe comparable populations.

Outcome Scales and Fair Interpretation

Metric values cannot be compared responsibly unless the outcome scale is understood. For a quantitative trait, a positive change of units applied consistently to both outcomes and predictions leaves \(R^2\) unchanged. If \(Y^*=c+dY\) and \(\hat{Y}^*=c+d\hat{Y}\), for \(d>0\), then both sums of squares are multiplied by \(d^2\):

The factor \(d^2\) cancels in their ratio, so \(R^2\) is unchanged. Absolute error measures, such as root mean squared error, do not have this invariance: when the outcome is expressed in units multiplied by \(d\), the error is also multiplied by \(d\). Thus \(R^2\) and an error measured in trait units provide complementary information.

Standardizing each group’s outcome separately can make within-group \(R^2\) values interpretable on a common dimensionless scale, but it does not make their absolute errors directly comparable in the original trait units. For binary outcomes, changing the case definition or prediction horizon changes the task itself; the resulting AUC, Brier score, and calibration summaries are not simply alternative measurements of an unchanged outcome. Nor should an \(R^2\) for a quantitative phenotype be treated as directly equivalent to an AUC for a binary diagnosis.

A Short Derivation: Why the Metrics Need Not Move Together

The distinct behavior follows from the definitions. For quantitative predictions, \(R^2_g\) is determined by the ratio of residual variation to total outcome variation. Holding the residual sum of squares fixed while increasing the total sum of squares increases \(R^2_g\), as in the first example. The metric therefore incorporates the group’s outcome variability.

For binary predictions, AUC depends on pairwise orderings, while Brier score depends on the numerical distance between each probability and its observed outcome. Replacing probabilities with another strictly increasing set of values can preserve every ordering and hence AUC, while changing each squared error. Calibration is likewise about the relationship of probability values to outcome frequencies, not only whether cases tend to rank above controls. Consequently, there is no general rule that an ancestry group with lower AUC must also have a worse Brier score, or that a group with lower \(R^2\) must have poorer calibration.

Practical Comparison and Common Pitfall

Before interpreting a difference between groups, verify that both estimates concern the intended prediction task. Check that continuous traits use the same measurement units and phenotype definition, and that binary outcomes use the same case criteria and time horizon. Confirm that the evaluation samples represent the target groups, or that the analysis accounts for sampling differences relevant to the chosen metric. Then report metrics by name and in context: an \(R^2\) difference is about relative squared error against group outcome variation; an AUC difference is about case-control ranking; a calibration difference concerns predicted risk versus observed outcomes; and a Brier difference concerns squared probability errors with contributions weighted by the evaluated sample composition.

Common mistake: “Group A has a higher AUC, so its risk predictions are better calibrated and its explained variance must also be higher.” This conclusion does not follow. AUC describes ordering for a binary outcome; calibration describes probability agreement; \(R^2\) describes a quantitative prediction-error ratio. Each must be measured and interpreted separately. A metric can support a claim only about the property it summarizes.