Standardizing Score Performance to a Target Population

Why Standardize Score Performance?

A polygenic score can be evaluated in a recruited cohort whose composition differs from the population where the score is intended to be used. For example, participation may be more common in one age group, ancestry group, or geographic stratum than another. If prediction performance differs across those strata, a summary calculated directly in the recruited cohort describes that cohort’s mixture—not necessarily the target population’s mixture.

Standardization addresses this compositional difference by combining stratum-specific performance estimates using the target population’s stratum proportions. It does not, by itself, remove bias within a stratum or establish that the recruited participants represent all target individuals within that stratum. Those conditions must be considered separately.

The recruited-cohort summary instead uses the recruited proportions: \(\theta_R=\sum_z r_z\theta_z\), when the quantity is an average of individual-level stratum values. If \(q_z\) and \(r_z\) differ, these summaries can differ even when performance within every stratum is estimated correctly. The target proportions may come from a population register, census, or other suitable external information; they should not be substituted with recruited proportions simply because those are easier to calculate.

What Standardization Requires

For each stratum with positive target proportion, the evaluation data need to provide information about performance in that stratum. A central transport assumption is that, conditional on the observed strata used for standardization, the relevant outcome and score distributions in recruited participants represent those in the target population. The assumption concerns the quantities being estimated: for example, the distribution of outcomes and predictions within a stratum for calibration, and the distributions among cases and controls for discrimination.

This assumption can fail if participation is associated with score or outcome even among people in the same recorded stratum. Standardizing on a broad category may also leave important compositional differences unaddressed. The target and evaluation data must use compatible score definitions, outcome definitions, follow-up periods, and measurement procedures. Finally, every stratum with \(q_z>0\) must be represented in the evaluation data; otherwise its performance cannot be estimated from those data without additional assumptions.

This identity gives the basic justification: the target-population mean of a quantity \(g\) is the sum of its stratum-specific means, weighted by target stratum proportions. If the recruited data estimate each target stratum-specific mean, replacing \(E_T[g\mid Z=z]\) with its estimate gives a direct standardization estimator. For an average metric such as a mean prediction error or Brier score, this is the required calculation.

A mean calibration gap is useful but limited: positive and negative errors at different risk levels can cancel. Calibration can also be summarized by comparing observed and predicted risks across risk groups or by fitting a calibration model. Whatever summary is chosen, target standardization must use the appropriate target-weighted stratum-specific quantity; weighting an already-combined recruited-cohort statistic cannot generally recover the target result.

Standardizing an Average Prediction Error

The Brier score for a binary outcome is the mean squared difference between the outcome \(Y\), coded 0 or 1, and the predicted risk \(\hat p\). Smaller values indicate predictions closer to the observed outcomes on average, though the score reflects both calibration and discrimination. Because it is an average of individual squared errors, it can be standardized by weighting stratum-specific Brier scores by target stratum proportions.

Target-Population Discrimination

The area under the receiver operating characteristic curve (AUC) is the probability that a randomly selected case has a higher score than a randomly selected control, with half credit for a tie. A target-population AUC compares cases and controls drawn from the target population. Its stratum weights are therefore not generally just the target population proportions: cases and controls can have different stratum distributions when outcome prevalence varies across strata.

The double sum includes comparisons between cases and controls from different strata. The within-stratum AUC \(A_{zz}\) measures ranking when both members of a case-control pair come from stratum \(z\). A target-weighted average of these within-stratum AUCs describes a different quantity from the target-population AUC unless cross-stratum comparisons are also accounted for or special conditions make them equivalent.

Practical Interpretation and a Common Pitfall

Standardization changes the population mixture used to summarize performance; it does not alter a person’s score or the conditional performance estimates. A target-standardized result answers a population-specific predictive question. It is not evidence that the score causes the outcome, and it does not show that performance will transport to unmeasured or unsupported strata.

A frequent mistake is to calculate the target AUC as \(\sum_zq_zA_{zz}\) and assume this is the target-population AUC. That calculation averages within-stratum ranking performance, but it omits cross-stratum case-control pairs and uses population rather than case and control stratum weights. In the worked example it gives \(0.40(0.75)+0.60(0.75)=0.75\), not the target AUC of approximately 0.669. The correction is to define the target AUC as a case-control comparison in the target population, calculate \(c_z\) and \(d_z\), and include every \(A_{zk}\) term. If the intended estimand is instead a weighted average of within-stratum AUCs, label it as such rather than calling it the overall target-population AUC.

Uncertainty in standardized estimates reflects uncertainty in the stratum-specific performance estimates and, when estimated rather than known, the target proportions and prevalences. An interval calculation should preserve the dependence among observations and all estimated components, for example by resampling participants within the relevant sampling strata or using an appropriate design-based method. Reporting only a point estimate can obscure substantial uncertainty, especially when some strata contain few cases or controls.