Calibration and Discrimination as Separate Properties

Two Different Questions About a Risk Prediction

A binary-risk model assigns each subject a probability of an event, such as a complication within 30 days. Two questions naturally follow. Are the probabilities numerically trustworthy? And do the predictions rank subjects who experience the event above those who do not? The first concerns calibration; the second concerns discrimination. They are related aspects of prediction, but neither one answers the other question.

This distinction matters in practice. A hospital might need accurate absolute risks to plan staffing, while a screening program might use risk rankings to decide whom to examine first. A model can be useful for one purpose and inadequate for the other. Assessing both properties gives a fuller account of probability predictions than either alone.

Definition: Let $Y$ be a binary outcome taking values 0 and 1, and let $P\in[0,1]$ be a predicted event probability. The prediction is calibrated if $E[Y\mid P]=P$ almost surely. Equivalently, among subjects receiving prediction $p$, the event probability is $p$, wherever that conditional probability is defined. Discrimination describes how well the predictions rank subjects with $Y=1$ above subjects with $Y=0$.

Calibration is a statement about the relationship between predicted probabilities and outcomes. In a finite data set, a prediction value may occur for only a few subjects, or not be repeated at all; its observed event fraction will then be noisy or undefined. Thus exact equality of a small group’s observed fraction to its predicted probability is not required for a model to be calibrated in the population. The definition is a conditional-probability statement, not a demand that every finite sample match its predictions exactly.

A common summary of discrimination is the area under the receiver operating characteristic curve (AUC). Its probability interpretation makes the ranking question precise.

Definition: Let $P_1$ and $P_0$ be independent predictions drawn, respectively, from the prediction distributions among subjects with $Y=1$ and $Y=0$. Provided both outcome classes have positive probability, $$ \operatorname{AUC} = P(P_1>P_0)+\frac{1}{2}P(P_1=P_0). $$ Thus the AUC is the probability that a randomly selected case receives a higher prediction than a randomly selected noncase, with a tie counted as half a win. It lies between 0 and 1; $1/2$ is the value for uninformative ranking.

Calibration Does Not Guarantee Discrimination

Suppose every subject receives the same prediction: the event prevalence. The prediction can be exactly calibrated, because the event probability among all subjects is that prevalence. But because every prediction is tied, no case is ranked above any noncase. The following result formalizes both points.

Result: If $P=\pi$ almost surely, where $\pi=P(Y=1)$ and $0<\pi<1$, then the prediction is calibrated and its AUC is $1/2$.

Check calibration.
Since $P$ is constant, $E[Y\mid P]=E[Y]=P(Y=1)=\pi=P$. This is precisely the calibration condition.

Check discrimination.
A case and a noncase both receive $\pi$, so $P_1=P_0$ with probability 1. The AUC is therefore $0+\tfrac12(1)=\tfrac12$. $\square$

Worked Example: Calibrated but Unable to Rank Risk

In a population of 100 patients, 30 experience the event and 70 do not. A model predicts a risk of 0.30 for every patient.

Calibration. Among the 100 patients receiving 0.30, the event fraction is $30/100=0.30$. In the population represented by these data, the common prediction matches the event prevalence.

Discrimination. Every case has prediction 0.30, and every noncase also has prediction 0.30. Therefore every case–noncase pair is tied, and $\operatorname{AUC}=0+\tfrac12(1)=0.50$.

The prediction communicates the overall event rate but cannot distinguish which patients are more likely to experience the event. Calibration alone does not establish useful ranking.

The same logic applies beyond a single constant prediction: calibration constrains event probabilities at prediction values, but it does not require those values to vary across subjects. A model that gives everyone the prevalence can be calibrated even though it cannot separate higher-risk from lower-risk patients.

Discrimination Does Not Guarantee Calibration

Rankings depend on the order of predictions, whereas calibration depends on their numerical values. A strictly increasing transformation changes the values without changing their order. It therefore preserves discrimination, but it need not preserve calibration.

Result: If $g$ is a strictly increasing function on the prediction range, replacing $P$ by $g(P)$ leaves the AUC unchanged, provided the transformed scores are used to rank cases and noncases. In contrast, calibration of $P$ does not in general imply calibration of $g(P)$.

To prove the AUC claim, take any independent case–noncase pair $(P_1,P_0)$. Strict increase gives $P_1>P_0$ if and only if $g(P_1)>g(P_0)$, and $P_1=P_0$ if and only if $g(P_1)=g(P_0)$. Thus both the strict-win probability and the tie probability in the AUC definition are unchanged. Calibration can change because the transformed numerical value need not equal the conditional event probability: even if $E[Y\mid P]=P$, there is no general identity $E[Y\mid g(P)]=g(P)$. When $g$ is one-to-one, conditioning on $g(P)$ contains the same information as conditioning on $P$, so the conditional mean remains $P$, not necessarily $g(P)$. $\square$

Worked Example: Good Ranking With Distorted Probabilities

Suppose there are two risk groups, each containing 10 patients. In the lower-risk group, 2 patients experience the event; in the higher-risk group, 8 do. First assign predictions 0.20 to the lower group and 0.80 to the higher group.

Calibration of the original predictions. The event fractions are $2/10=0.20$ and $8/10=0.80$, so the predictions match the observed fractions in both groups.

Discrimination. There are 10 cases and 10 noncases, hence 100 case–noncase pairs. The 8 higher-group cases each outrank the 8 lower-group noncases, giving 64 concordant pairs. There are 16 ties within the lower group and 16 ties within the higher group; the remaining $2\times2=4$ pairs are discordant. Therefore $$ \operatorname{AUC} = \frac{64+\tfrac12(16+16)}{100} = \frac{80}{100} = 0.80. $$

Now replace each prediction $p$ by $g(p)=0.10+0.80p$. The two predictions become 0.26 and 0.74. This transformation is strictly increasing, so the ranking and AUC remain unchanged. But the observed event fractions are still 0.20 and 0.80, not 0.26 and 0.74. The transformed predictions are therefore miscalibrated in these groups despite retaining the same discrimination.

AUC as a Ranking Probability

AUC is often easier to interpret as a pairwise probability than as a feature of a curve. For example, an AUC of 0.80 means that a randomly drawn case has a higher prediction than a randomly drawn noncase in 80% of comparisons when there are no ties; with ties, the half-credit convention applies. This interpretation depends on the case and noncase prediction distributions, not on whether a predicted value of 0.70 corresponds to a 70% event probability.

Worked Example: AUC of 0.9231 Without Calibration

Consider 13 cases and 13 noncases. Among the cases, 12 receive score 0.90 and 1 receives score 0.10. Every noncase receives score 0.50. Treat these scores as probability predictions for purposes of assessing calibration.

Compute the AUC. There are $13\times13=169$ case–noncase pairs. Each of the 12 cases with score 0.90 outranks all 13 noncases, producing $12\times13=156$ concordant pairs. The one case with score 0.10 loses to all 13 noncases, producing 13 discordant pairs. There are no ties. Hence $$ \operatorname{AUC} = \frac{156}{169} = \frac{12}{13} \approx 0.9231. $$

Check calibration. The score 0.90 occurs only among cases in this data set, so its observed event fraction is $1$, not 0.90. The score 0.10 also occurs only among cases, so its observed event fraction is $1$, not 0.10. The score 0.50 occurs only among noncases, so its observed event fraction is $0$, not 0.50. These observed fractions do not match the assigned probability values. The example has strong ranking but does not have exact groupwise calibration in the data.

This is a finite-sample illustration, not proof that the underlying population conditional probabilities equal those observed fractions. Its point is that a high AUC is a ranking statement; it does not by itself certify numerical accuracy of risk predictions.

Using Both Properties in Assessment

Calibration and discrimination answer complementary questions. Calibration assesses whether predictions are on the right probability scale; discrimination assesses whether predictions order cases above noncases. A useful interpretation reports them separately rather than treating either as a substitute for the other.

Calibration	Discrimination	Interpretation
Good	Good	Predicted probabilities are reasonably aligned with event frequencies and also rank outcomes effectively.
Good	Poor	Probabilities may represent the overall or local event rates, but rankings offer little separation.
Poor	Good	Rankings may identify relatively higher-risk subjects, while the reported probability values need correction or cautious interpretation.
Poor	Poor	Neither probability accuracy nor ranking is satisfactory for the intended use.

A further mathematical distinction is that calibration implies a useful population-average identity. Taking expectations on both sides of $E[Y\mid P]=P$ and applying the law of total expectation gives $$ E[P] = E\{E[Y\mid P]\} = E[Y] = P(Y=1). $$ Thus calibrated predictions have mean equal to the event prevalence. The converse does not hold: matching the mean prediction to prevalence checks only one average, not the conditional relationship at different prediction values. Likewise, neither this mean identity nor a favorable AUC establishes calibration across the range of predicted risks.

Common Mistake

Wrong reasoning: “The model has a high AUC, so its predicted probabilities are accurate.” This confuses ordering with probability scale. A strictly increasing transformation can preserve every ranking and hence preserve AUC while changing the probabilities enough to make them poorly calibrated.

Correction: State what each diagnostic supports. AUC describes case–noncase ranking; calibration diagnostics assess agreement between predicted risks and observed event frequencies. Neither finding should be reported as evidence for the other property. The appropriate balance depends on how the predictions will be used.

Key takeaway: Calibration concerns probability values; discrimination concerns ranking. A model can have either property without the other, so probability predictions should be interpreted by assessing both questions separately.

Check Your Understanding

Use the definitions and examples above to answer these questions.

A model assigns every patient the population event prevalence. Under what condition is it calibrated, and what is its AUC?
In the AUC definition, why does a tied case–noncase pair receive half credit?
Explain why a strictly increasing transformation preserves AUC but may change calibration.
A model has AUC 0.90. What does this say about case–noncase rankings, and what does it not establish about the numerical risk predictions?
Why does a mean predicted risk equal to the event prevalence not, by itself, establish calibration?