What a Worked Validation Analysis Evaluates
A polygenic score is not validated simply by showing that it separates people with and without an outcome in the data used to develop it. The relevant question is how well a prespecified prediction procedure performs for new people. That procedure might use a polygenic score (PGS) alone or combine it with other predictors; it must specify how the score becomes a predicted risk. Validation then compares those predictions with outcomes observed in a separate cohort.
This tutorial uses a binary outcome, coded \(Y_i=1\) when person \(i\) experiences the outcome and \(Y_i=0\) otherwise. Let \(\hat{p}_i\) be the predicted probability of the outcome for that person. The hat indicates a prediction, not the observed outcome. The validation cohort must be independent of the data used to select, tune, or fit the prediction procedure. The score, risk mapping, outcome definition, and any decision threshold should be fixed before examining validation outcomes. Otherwise, apparent performance can benefit from information that would not be available when predicting for new people.
The metrics below answer different questions. AUC assesses ranking; calibration assesses agreement between predicted risks and observed outcomes; the Brier score summarizes squared probability errors; and predictive values describe the outcomes among people classified above or below a particular threshold. None of these metrics alone establishes that using a score improves health outcomes.
Worked Example: AUC, Brier Score, and a Risk Threshold
Worked Example: A Six-Person Hypothetical Validation Set
Suppose a locked PGS-based risk model is applied to six unrelated people in a hypothetical independent cohort. The table gives each person’s predicted risk and observed outcome. For illustration, use a prespecified threshold of 0.50: predictions at or above 0.50 are classified as positive.
| Person | Predicted risk, \(\hat{p}_i\) | Outcome, \(Y_i\) |
|---|---|---|
| A | 0.10 | 0 |
| B | 0.20 | 1 |
| C | 0.30 | 0 |
| D | 0.40 | 1 |
| E | 0.60 | 0 |
| F | 0.80 | 1 |
AUC. There are three cases (B, D, F) and three non-cases (A, C, E), so there are \(3\times3=9\) case–non-case pairs. A pair is concordant when the case has the higher predicted risk. Case B outranks one non-case; D outranks two; and F outranks all three. Thus,
The model ranks a randomly selected case above a randomly selected non-case in about 66.7% of such pairs in this small cohort. This is a ranking interpretation, not a statement that 66.7% of individual predictions are correct. There are no ties here; a tied case–non-case pair would count as one-half of a concordant pair.
Brier score. For a binary outcome, the score is the mean of \((\hat{p}_i-Y_i)^2\). Substituting the six predictions and outcomes gives
The Brier score is on the squared probability scale, with lower values indicating smaller average squared errors. A value of 0.250 does not, on its own, identify whether error comes from poor ranking, poor calibration, or both.
Threshold-specific values. At the fixed threshold of 0.50, E and F are positive, while A, B, C, and D are negative. F is a true positive, E a false positive, B and D false negatives, and A and C true negatives. Therefore \(TP=1\), \(FP=1\), \(FN=2\), and \(TN=2\). The predictive values are
In this hypothetical set, half of the people classified positive had the outcome, and half of those classified negative did not. These estimates are very imprecise because the cohort is tiny; the arithmetic illustrates the definitions, not reliable performance estimates.
Calibration Intercept and Slope
Calibration is assessed by relating observed binary outcomes to the predicted risks. Convert each risk to its log odds, or linear predictor, \(\eta_i=\operatorname{logit}(\hat{p}_i)=\log\{\hat{p}_i/(1-\hat{p}_i)\}\). A commonly reported calibration model is
Here \(a\) is the calibration intercept and \(b\) is the calibration slope in this joint model. Ideal calibration has \(a=0\) and \(b=1\). An intercept below zero often indicates that the model’s risks are too high overall, while an intercept above zero often indicates risks that are too low. A slope below one indicates predictions that are too extreme on the log-odds scale; a slope above one indicates predictions that are not extreme enough, when the joint model is an appropriate summary.
Calibration-in-the-large is a closely related but distinct intercept diagnostic. It is estimated with the predicted linear predictor held fixed as an offset:
Thus, \(a_{\mathrm{CITL}}\) assesses a common shift in log odds while fixing the slope at one. It need not equal the intercept \(a\) from the joint model, where the slope is also estimated. A clear report states which intercept definition is being used.
Worked Example: A Calibration Intercept and Slope
Consider another hypothetical independent cohort of 34 people, with 17 assigned predicted risk 0.20 and 17 assigned predicted risk 0.80. Among the first group, one person has the outcome; among the second, 16 do. This deliberately compact grouped example makes the calibration calculations transparent.
The predicted log odds are \(\eta_L=\log(0.20/0.80)=-\log(4)\) and \(\eta_H=\log(0.80/0.20)=\log(4)\). The observed outcome proportions are \(1/17\) and \(16/17\), whose log odds are \(-\log(16)\) and \(\log(16)\), respectively. Since \(\log(16)=2\log(4)\), the two observed log odds equal \(2\eta_L\) and \(2\eta_H\). Therefore the joint calibration model fits with
This is also the maximum-likelihood fit: in each prediction group, the fitted event probability is the observed proportion, so the likelihood score for that group’s fitted log odds is zero. The two distinct predictor values then identify the intercept and slope, yielding the values above. The slope of 2 indicates predictions that are not sufficiently spread out on the log-odds scale in this example.
For calibration-in-the-large, the observed event count is \(1+16=17\). With the offset unchanged, the sum of predicted event probabilities is \(17(0.20)+17(0.80)=17\). Consequently \(a_{\mathrm{CITL}}=0\) satisfies the intercept score equation: observed events equal the fitted total. This example shows why a satisfactory overall event count does not guarantee a satisfactory calibration slope.
Why the Metrics Answer Different Questions
AUC depends on the ordering of predicted risks, not their absolute values. Any strictly increasing transformation of predictions leaves the ranking—and therefore AUC—unchanged, even if the resulting risks are systematically too high or too low. Calibration checks the probability scale that AUC leaves aside. The Brier score uses both the predictions and outcomes, so it reflects probability accuracy without isolating a single aspect of performance.
For a cohort of \(n\) people, the Brier score is
Each prediction is compared with the outcome that occurred: for a case, the contribution is \((1-\hat{p}_i)^2\); for a non-case, it is \(\hat{p}_i^2\). The score is bounded between 0 and 1 for individual binary outcomes, with 0 representing perfect predictions. Comparisons across cohorts should be cautious because outcome prevalence and case mix affect the score.
Predictive values depend on the chosen threshold and the cohort’s outcome prevalence. For a threshold \(c\), classify person \(i\) as positive when \(\hat{p}_i\geq c\); otherwise classify them as negative. The resulting counts define
These definitions use the number classified positive or negative as their respective denominators. PPV is not sensitivity, which is \(TP/(TP+FN)\); NPV is not specificity, which is \(TN/(TN+FP)\). A threshold should be justified for the intended use rather than chosen after inspecting which value gives the most appealing result.
Worked Example: Predictive Values at a Fixed Threshold
Suppose a prespecified threshold is applied in a hypothetical cohort of 200 people. The outcome prevalence is 10%, so 20 people are cases and 180 are non-cases. At this threshold, assume the model identifies 80% of cases as positive and 90% of non-cases as negative. The corresponding counts are \(TP=0.80(20)=16\), \(FN=20-16=4\), \(TN=0.90(180)=162\), and \(FP=180-162=18\). The predictive values are
At this threshold in this hypothetical cohort, 47.1% of those classified positive had the outcome, while 97.6% of those classified negative did not. The same sensitivity and specificity can produce different predictive values in a population with a different prevalence, because the numbers of cases and non-cases entering the denominators change.
Assumptions and a Practical Analysis Sequence
The calculations are meaningful only when the validation design supports their interpretation. Before computing metrics, check that the outcome and prediction refer to the same defined target and time horizon; that each person’s prediction was generated without using their validation outcome; and that the score and risk mapping were fixed independently of the validation results. For an external validation, also report who entered the cohort and how missing predictors or outcomes were handled. A metric among people with complete data describes that analyzed group unless a justified method addresses missingness.
The formulas above describe empirical performance in the analyzed validation sample. For inference to a wider target population, uncertainty matters: AUC, calibration estimates, Brier score, and predictive values all vary across samples. The estimate alone does not convey that uncertainty. The validation unit and sampling design also matter; for example, treating related individuals as independent can understate uncertainty. These issues do not change the metric definitions, but they affect how precisely a result can be generalized.
Specify the score, risk mapping, outcome, prediction horizon, validation eligibility rules, and any threshold without using validation outcomes to tune them.
Confirm that each analyzed person has a prediction and a correctly coded binary outcome, and describe exclusions and missingness.
Report AUC for ranking, calibration intercept and slope for risk agreement, Brier score for squared probability error, and threshold-specific counts and predictive values where relevant.
State the cohort, outcome, threshold, units or scales, and uncertainty. Do not treat performance in one cohort as proof of performance in every population or clinical setting.
Common Mistakes
- Calling a high AUC good calibration. AUC measures ranking and can remain unchanged when predicted risks are distorted. Examine calibration separately.
- Confusing a calibration intercept with the slope. The intercept concerns a shift in log odds; the slope concerns the spread of predictions on the log-odds scale. State whether the intercept is calibration-in-the-large or comes from the joint model.
- Interpreting PPV as sensitivity. PPV divides true positives by everyone classified positive; sensitivity divides true positives by all cases. Their denominators answer different questions.
- Comparing predictive values without considering prevalence. The same threshold and sensitivity and specificity do not guarantee the same PPV or NPV in a population with different outcome prevalence.
- Reporting a validation result after tuning on the same outcomes as if it were independent. Choosing a threshold or adjusting the risk mapping using validation outcomes makes the reported assessment optimistic for new data. Use a genuinely independent evaluation for the final performance estimate.
Check Your Understanding
Use the definitions and examples above to answer the following questions.
- In a validation set with 12 cases and 18 non-cases, how many case–non-case pairs contribute to the AUC calculation?
- What does a calibration slope below one indicate about the spread of predictions on the log-odds scale?
- At a fixed threshold, 24 people are classified positive and 9 of them have the outcome. Calculate the PPV and state its denominator.
- Why can two populations with the same sensitivity and specificity have different PPVs?
- Can a model have an unchanged AUC after its risk predictions are transformed but have worse calibration? Explain briefly.