Calibration Diagnostics for Multiclass Predictions

Why Calibration Diagnostics Need More Than One View

A classifier can assign a probability to every class and still be poorly calibrated. For example, among cases assigned probability 0.7 to a particular class, the class should occur about 70% of the time if those probabilities are calibrated. In multiclass problems, that check can be made separately for each class. It does not, by itself, verify that the entire probability vector is calibrated.

Calibration diagnostics summarize this relationship between stated probabilities and observed outcomes. They are distinct from classification error: a classifier can choose the most frequent class correctly while giving unreliable probabilities, or give well-calibrated probabilities while making some classification errors. A useful diagnostic therefore reports how predictions compare with outcomes, while making clear what information has been grouped or averaged.

Definition: Let $Y\in\{1,\ldots,K\}$ be the class label, and let $P=(P_1,\ldots,P_K)$ be the predicted probability vector, with $P_k\geq 0$ and $\sum_{k=1}^K P_k=1$. The predictions are classwise calibrated if, for every class $k$, $$ E[\mathbf{1}\{Y=k\}\mid P_k]=P_k \quad \text{almost surely}. $$ They are vector calibrated if, for every class $k$, $$ E[\mathbf{1}\{Y=k\}\mid P]=P_k \quad \text{almost surely}. $$ Vector calibration conditions on the full probability vector; classwise calibration conditions only on the probability assigned to the class being checked.

These definitions describe population properties, not what a finite reliability diagram proves. In practice, diagnostics are estimated from cases with observed outcomes and predicted probabilities, ideally from data not used to fit the classifier. The empirical estimates are subject to sampling variability, especially when a class or a probability range occurs infrequently.

Building a Classwise Reliability Diagram

For a chosen class $k$, divide the possible values of $P_k$ into bins $B_1,\ldots,B_J$. For each bin containing at least one evaluation case, calculate the average predicted probability and the observed fraction belonging to class $k$. Plot the observed fraction vertically against the average prediction horizontally. Perfect bin-level agreement places the point on the diagonal.

Definition: For evaluation observations $i=1,\ldots,n$, let $Y_i$ be the observed class and $P_{ik}$ the predicted probability for class $k$. For bin $B_j$, define $I_{jk}=\{i:P_{ik}\in B_j\}$ and $n_{jk}=|I_{jk}|$. If $n_{jk}>0$, the bin's mean prediction, observed frequency, and signed gap are $$ \bar P_{jk}=\frac{1}{n_{jk}}\sum_{i\in I_{jk}}P_{ik},\qquad \bar Y_{jk}=\frac{1}{n_{jk}}\sum_{i\in I_{jk}}\mathbf{1}\{Y_i=k\},\qquad D_{jk}=\bar Y_{jk}-\bar P_{jk}. $$ A positive gap means the class occurred more often than predicted in that bin; a negative gap means it occurred less often.

A reliability diagram is classwise: a multiclass problem generally calls for one diagram per class. One can also summarize the bin gaps, but the summary depends on the chosen bins and weights. For instance, a frequency-weighted absolute calibration error for class $k$ is $$ \sum_{j:n_{jk}>0}\frac{n_{jk}}{n}|D_{jk}|. $$ This is a descriptive statistic for a particular partition and sample, not a bin-independent measure of population calibration. Taking absolute values prevents positive and negative bin gaps from cancelling in this summary, but it does not remove the effects of bin choice or sampling noise.

Worked Example: Classwise Calibration Without Vector Calibration

Suppose four equally likely prediction vectors are made, with the following predicted probabilities and actual conditional class distributions. Each row of the actual-distribution column is $P(Y=1),P(Y=2),P(Y=3)$ conditional on that prediction vector.

Prediction vector $P$	Actual conditional distribution of $Y$
(0.50, 0.30, 0.20)	(0.54, 0.28, 0.18)
(0.50, 0.20, 0.30)	(0.46, 0.22, 0.32)
(0.40, 0.30, 0.30)	(0.40, 0.32, 0.28)
(0.60, 0.20, 0.20)	(0.60, 0.18, 0.22)

First check that each actual row is a valid probability distribution: its entries are nonnegative and the four row sums are $0.54+0.28+0.18=1$, $0.46+0.22+0.32=1$, $0.40+0.32+0.28=1$, and $0.60+0.18+0.22=1$.

For class 1, the prediction $P_1=0.50$ occurs in the first two rows. Since those rows are equally likely, the conditional class-1 frequency is $(0.54+0.46)/2=0.50$. At $P_1=0.40$, the frequency is $0.40$; at $P_1=0.60$, it is $0.60$. Thus class 1 is calibrated conditional on $P_1$.

For class 2, $P_2=0.30$ occurs in rows one and three, where the average class-2 frequency is $(0.28+0.32)/2=0.30$. At $P_2=0.20$, the average across rows two and four is $(0.22+0.18)/2=0.20$. For class 3, $P_3=0.20$ occurs in rows one and four, giving $(0.18+0.22)/2=0.20$; $P_3=0.30$ occurs in rows two and three, giving $(0.32+0.28)/2=0.30$. All three classes are therefore classwise calibrated.

But vector calibration fails. For example, when $P=(0.50,0.30,0.20)$, the conditional probability of class 1 is $0.54$, not $0.50$. The other vectors also have conditional distributions that differ from their predictions. Averaging across different vectors with the same class-specific coordinate makes the classwise checks agree, even though conditioning on the full vector does not.

What Binning Preserves—and What It Can Hide

A bin combines observations with potentially different predicted probabilities. Its plotted horizontal coordinate is the mean prediction within that bin, not a probability value shared by every observation in it. Its vertical coordinate is the corresponding average outcome. A point on the diagonal therefore establishes agreement of these two averages for that bin, but not agreement at every probability value inside it.

Result: Within any nonempty bin $B$ for class $k$, the population-level observed frequency minus mean prediction is $$ E[\mathbf{1}\{Y=k\}-P_k\mid P_k\in B]. $$ Consequently, positive and negative deviations at different values of $P_k$ can cancel when they are pooled into one bin.

To verify the identity, use linearity of conditional expectation: $$ E[\mathbf{1}\{Y=k\}\mid P_k\in B]-E[P_k\mid P_k\in B] =E[\mathbf{1}\{Y=k\}-P_k\mid P_k\in B]. $$ A zero bin gap says that this conditional average is zero. It does not say that the conditional average given each finer value of $P_k$ is zero.

Worked Example: A Coarse Bin Hides Classwise Miscalibration

Consider two equally likely prediction situations in a three-class problem. Their predicted vectors are $(0.20,0.40,0.40)$ and $(0.80,0.10,0.10)$; their actual conditional class distributions are $(0.40,0.30,0.30)$ and $(0.60,0.20,0.20)$. Each actual row sums to one.

For class 1, use a single bin containing both predicted probabilities. The mean prediction is $(0.20+0.80)/2=0.50$, and the observed frequency is $(0.40+0.60)/2=0.50$. The diagram shows no gap. But at $P_1=0.20$, the class-1 frequency is $0.40$, and at $P_1=0.80$, it is $0.60$; neither value matches its prediction.

The same pooling can mask discrepancies for the other classes. For class 2, the mean predicted probability is $(0.40+0.10)/2=0.25$, and the mean observed frequency is $(0.30+0.20)/2=0.25$. For class 3, these averages are also $0.25$ and $0.25$. A diagram made with one broad bin for each class would show agreement, despite the mismatches within the bins. Narrower bins would expose these discrepancies, although they would contain fewer observations and have noisier observed frequencies.

The tension is fundamental: coarse bins stabilize estimates by pooling more observations, while fine bins retain more detail but may be unstable. A reliability diagram should therefore be read together with its bin boundaries and bin counts. If conclusions change substantially under reasonable alternative partitions, the diagnostic is sensitive to binning and should not be summarized as a definitive calibration verdict.

Class Imbalance and the Meaning of an Overall Summary

When classes are imbalanced, most observations may belong to common classes or to probability ranges that occur often. A frequency-weighted summary then gives those regions more influence. This weighting can be appropriate for describing average performance over the evaluation population, but it can make large errors in rare classes or small subgroups appear unimportant. Conversely, a rare-class reliability point may be highly variable because it is based on few cases.

Worked Example: An Overall Match Conceals a Rare-Group Error

Consider 1,000 predictions for three classes. For 990 observations, the predicted vector is $(1/198,197/396,197/396)$; among these observations, 495 outcomes are class 2 and 495 are class 3, with none in class 1. For the other 10 observations, the predicted vector is $(0.50,0.25,0.25)$, and all 10 outcomes are class 1.

Both prediction vectors are valid: $1/198+197/396+197/396=2/396+394/396=1$, and $0.50+0.25+0.25=1$. The overall mean predicted class-1 probability is $(990/1000)(1/198)+(10/1000)(0.50)=0.005+0.005=0.01$. The observed class-1 frequency is $10/1000=0.01$.

For class 2, the overall mean prediction is $(990/1000)(197/396)+(10/1000)(0.25)=0.4925+0.0025=0.495$, matching its observed frequency $495/1000=0.495$. Class 3 has the same overall mean prediction and observed frequency. Thus the overall predicted class proportions match the observed class proportions exactly.

That match does not describe the two groups well. In the group of 990, the class-1 probability is about $0.00505$, but the observed class-1 frequency is zero. In the group of 10, its predicted probability is $0.50$, but the observed frequency is 1. A single broad class-1 bin containing all observations would show the matching values 0.01 and 0.01, concealing both group-specific discrepancies. The rare group also supplies only 10 observations, so its observed frequency is especially uncertain as an estimate of future performance.

It is useful to report class-specific diagrams and counts rather than relying only on a pooled summary. A classwise, frequency-weighted summary answers a population-weighted question; an unweighted average across classes gives each class equal influence instead. Neither weighting is automatically correct for every purpose. The choice should match the intended interpretation, and the underlying class counts should remain visible.

Two Calibration Implications

Vector calibration is a stronger condition than classwise calibration. The implication follows by conditioning on a coordinate of the full vector.

Theorem: If $P$ is vector calibrated, then it is classwise calibrated.

Proof. Fix a class $k$. Since $P_k$ is a function of $P$, the tower property of conditional expectation gives $$ E[\mathbf{1}\{Y=k\}\mid P_k] =E\!\left[E[\mathbf{1}\{Y=k\}\mid P]\mid P_k\right] =E[P_k\mid P_k] =P_k. $$ This is exactly classwise calibration for class $k$, and the argument applies to every class. The converse does not hold, as the four-vector example above demonstrates. $\square$

A second implication concerns what a reliability point can establish. If a bin is calibrated in the population, its observed frequency equals its mean prediction within that bin. This is an averaged condition, not a guarantee of calibration at every score value inside the bin. The cancellation example demonstrates why a diagonal point should be interpreted at the resolution of the partition used to create it.

Common Mistake: Treating a Diagonal Plot as Proof

A common mistake is to reason, “The points lie on the diagonal, so the multiclass predictions are calibrated.” That conclusion is too strong. The points may come from broad bins, finite samples, or only one class's diagram. Even exact classwise calibration does not imply vector calibration, because the full vector can contain information about the outcome that is not captured by a single coordinate. The correction is to state what was checked: the classes, bins, counts, and whether the claim concerns empirical bin averages, classwise calibration, or vector calibration.

Key takeaway: Classwise reliability diagrams compare each class's predicted probabilities with its observed frequencies. Their conclusions are limited by sampling variability, binning, and weighting; they do not by themselves establish calibration conditional on the full probability vector.

Check Your Understanding

Use the definitions and calculations in this tutorial to answer the following.

In one sentence, distinguish classwise calibration from vector calibration.
A bin has mean predicted probability 0.35 and observed frequency 0.29 for class $k$. What is the signed gap, and how should its sign be interpreted?
Why can a broad bin have zero calibration gap even when its constituent probability values are miscalibrated?
What does a frequency-weighted calibration summary emphasize when classes are imbalanced?
Does vector calibration imply classwise calibration? State the conditional-expectation argument, or explain why not.