Prediction Metrics Under Case-Control Sampling

Why Case-Control Sampling Changes Some Metrics

A case-control validation sample can be useful when an outcome is rare: investigators deliberately include more cases than would occur in the target population. This design can preserve information about how a score ranks cases relative to controls. But the fraction of cases in the sample is then a design choice, not an estimate of population prevalence. Metrics that depend on prevalence, including calibrated risk and predictive values, can therefore be misleading if calculated directly from the sample.

The key is to separate two questions. Discrimination asks whether cases tend to receive higher scores than controls. Absolute-risk prediction asks whether a predicted probability corresponds to the chance of the outcome in a specified population. Simple outcome-dependent sampling can preserve the first while changing the second.

Definition: In simple outcome-dependent sampling, inclusion in the validation sample depends on outcome status \(Y\), but not on the score or other predictors within each outcome group. Write \(S=1\) for inclusion, and suppose \(P(S=1\mid Y=1,R)=s_1\) and \(P(S=1\mid Y=0,R)=s_0\) for every score value \(R\), where \(s_1\) and \(s_0\) are constant. The case fraction in the sample may differ from the target-population prevalence.

This assumption means the sampled cases represent the target population’s cases with respect to the score, and the sampled controls represent its noncases with respect to the score. It does not mean the sample represents the population’s overall outcome frequency. If selection within cases or controls also depends on score, site, severity, or another predictor, the results below need not hold without further adjustment.

Correcting Risks to a Target Prevalence

Let \(\pi=P(Y=1)\) denote the target-population prevalence, and let \(q=P(Y=1\mid S=1)\) be the case fraction in the sampled validation data. Suppose a probability \(p_s\) is calibrated for the sampled case-control distribution at a given score value. Because the ratio of case to control sampling probabilities is constant across score values, the sample changes the odds of the outcome by a constant factor. The probability must be corrected before it is interpreted as target-population risk.

\operatorname{logit}(p_{\mathrm{target}}) = \operatorname{logit}(p_s)+\operatorname{logit}(\pi)-\operatorname{logit}(q)

Here \(\operatorname{logit}(u)=\log\{u/(1-u)\}\). Equivalently, multiply the sample-calibrated odds by the target-to-sample prevalence odds ratio, then convert the resulting odds back to a probability. The correction requires the target prevalence to be known or estimated from suitable population data.

Worked Example: Correcting a Sample-Calibrated Risk

Suppose a hypothetical validation sample has a case fraction \(q=0.50\), because cases and controls were sampled in equal numbers. For a particular score value, the model gives a sample-calibrated probability \(p_s=0.80\). The intended target population has prevalence \(\pi=0.02\). What is the corresponding target-population probability?

The sample-calibrated odds are \(0.80/(1-0.80)=4\). The target-to-sample prevalence odds ratio is \((0.02/0.98)/(0.50/0.50)=0.02041\).

\text{Target odds}=4(0.02041)=0.08163, \qquad p_{\mathrm{target}}=\frac{0.08163}{1+0.08163}=0.07547.

Thus the corresponding target-population risk is about 7.55%, not 80%. The sample probability describes risk under the artificially balanced sample distribution; it is not a target-population probability until the prevalence shift is corrected.

The same adjustment has a useful interpretation for logistic calibration. If \(\eta\) is a model’s log-odds prediction and the target calibration relation is \(\operatorname{logit}P(Y=1\mid\eta)=a+b\eta\), outcome-dependent sampling adds a constant to the intercept. Under the stated sampling assumption and a correctly specified logistic calibration relation, the slope \(b\) is unchanged, while the intercept shifts. In particular, an ideal target-population intercept of zero becomes \(\operatorname{logit}(q)-\operatorname{logit}(\pi)\) in the sample; correcting the intercept restores target prevalence. If the calibration relation is misspecified, a fitted slope need not be invariant, because sampling changes the distribution of scores used in the fit.

Predictive Values Depend on Prevalence

At a chosen threshold, sensitivity is the probability of a positive result among cases, and specificity is the probability of a negative result among noncases. Positive predictive value (PPV) is the probability of being a case among people with a positive result; negative predictive value (NPV) is the probability of being a noncase among people with a negative result. The predictive values combine the threshold’s sensitivity and specificity with the prevalence in the population where the prediction will be used.

\mathrm{PPV} = \frac{\pi\,\mathrm{sensitivity}} {\pi\,\mathrm{sensitivity}+(1-\pi)(1-\mathrm{specificity})}, \qquad \mathrm{NPV} = \frac{(1-\pi)\,\mathrm{specificity}} {(1-\pi)\,\mathrm{specificity}+\pi(1-\mathrm{sensitivity})}

Worked Example: Recalculating Predictive Values

In a hypothetical balanced validation sample of 500 cases and 500 controls, a fixed threshold classifies 400 cases as positive and 450 controls as negative. The sensitivity is \(400/500=0.80\), and the specificity is \(450/500=0.90\). The sample PPV is \(400/(400+50)=0.8889\), while its NPV is \(450/(450+100)=0.8182\). These predictive values reflect the sample’s 50% case fraction.

Suppose the intended population prevalence is 2%. Using the same sensitivity and specificity, the target-population PPV is

\mathrm{PPV} = \frac{0.02(0.80)} {0.02(0.80)+0.98(1-0.90)} = \frac{0.016}{0.114} = 0.1404.

The target-population NPV is

\mathrm{NPV} = \frac{0.98(0.90)} {0.98(0.90)+0.02(1-0.80)} = \frac{0.882}{0.886} = 0.9955.

At this threshold, about 14.0% of positive results would be expected to be cases in the target population, and about 99.5% of negative results would be expected to be noncases, assuming the sensitivity and specificity transfer to that population. The much larger sample PPV is not evidence that the score performs differently; it reflects the sample’s deliberately high case fraction.

Which Discrimination Comparisons Remain Valid?

For a fixed score threshold, sensitivity and specificity are conditional on outcome status: sensitivity is \(P(R\geq t\mid Y=1)\), and the false-positive rate is \(P(R\geq t\mid Y=0)\). Simple outcome-dependent sampling leaves both conditional score distributions unchanged. Therefore, the sensitivity and specificity at that threshold, and the ROC curve traced out across thresholds, estimate their target-population counterparts under the sampling assumption.

The area under the ROC curve (AUC) is the probability that a randomly selected case receives a higher score than a randomly selected control, with ties conventionally counted as one half. Because simple outcome-dependent sampling preserves the score distribution within each outcome group, it preserves this case-control comparison and hence the AUC. Two scores can therefore be compared by their AUCs in the same appropriately sampled validation set, even if its case fraction is not the population prevalence. That comparison concerns ranking cases against controls, not whether either score gives calibrated population risks.

Worked Example: Interpreting an AUC Under Case-Control Sampling

In a hypothetical validation sample, there are 20 cases and 20 controls. Comparing every case score with every control score gives 336 concordant pairs out of \(20\times20=400\) pairs, with no ties. The estimated AUC is

\widehat{\mathrm{AUC}}=\frac{336}{400}=0.840.

If cases and controls were sampled independently of score within their outcome groups, this estimates the target population’s case-control ranking probability. The balanced sample does not make the AUC invalid merely because the population prevalence is lower. However, the value 0.840 says nothing by itself about target-population PPV, NPV, or whether predicted probabilities are calibrated.

Other metrics combine cases and noncases using their relative frequencies and so generally change with the case fraction. Examples include accuracy at a threshold, the Brier score, log loss, and precision-recall summaries. For instance, the Brier score is the mean squared difference between the outcome \(Y\) and predicted probability \(p\); its unweighted sample average reflects the sample’s case-control composition. Under simple outcome-dependent sampling, a target-prevalence version can instead combine the average loss among cases and the average loss among controls using weights \(\pi\) and \(1-\pi\), respectively. A calibration plot likewise needs target-population weighting or corrected risks if it is intended to describe target-population calibration.

Why the Sampling Assumption Matters

The preservation results depend on selection being outcome-dependent only. If, for example, cases with more severe disease are more likely to be recruited, or controls are selected differently across score values, then the sampled score distribution within an outcome group can differ from the target distribution. Sensitivity, specificity, and AUC may then be distorted as well as prevalence-dependent metrics. A design that samples by outcome and also by predictor may require appropriate sampling weights or a more explicit selection model.

A further distinction is between evaluating a fixed threshold and choosing one. Sensitivity and specificity measured at a threshold chosen in the validation data are still conditional performance measures, but the reported values may be optimistic if the same data were used to select that threshold. A threshold intended to represent a target risk or a target-population decision should be chosen or interpreted on the corrected target scale, not by treating the case-control sample fraction as population prevalence.

Key takeaway: Under simple outcome-dependent sampling, score distributions within cases and controls are preserved. This supports target-population interpretation of fixed-threshold sensitivity and specificity and of AUC, but not of raw sample predictive values, calibration, or other prevalence-dependent metrics. Correct risks using the target prevalence, and check that sampling did not also depend on predictors within outcome groups.

Common Mistake

Wrong reasoning: “The sample has equal numbers of cases and controls, so a positive prediction has a 50% chance of being a case.” The sample case fraction is set by ascertainment and does not establish the population chance.
Correction: A predictive value must use the prevalence of the intended target population, together with sensitivity and specificity. Likewise, a case-control sample can support an AUC estimate under the stated assumption, but a preserved AUC does not make its raw predicted probabilities or PPV population-calibrated.

Check Your Understanding

Answer each question by identifying the sampling assumption or population quantity that matters.

A study samples 70% cases, but the intended population prevalence is 4%. Which prevalence belongs in a target-population PPV calculation, and why?
Under simple outcome-dependent sampling, what feature of the design allows the sample AUC to estimate the target-population AUC?
A model is calibrated to a balanced case-control sample. What information is needed to correct its predicted probabilities to a target population?
Why might an unweighted Brier score from a case-control sample fail to represent target-population prediction error?
Give one example of a sampling scheme that could invalidate the claim that sensitivity and specificity are preserved.