Calibration and Proper Scores for Binary Predictions

Probability Forecasts and Their Evaluation

A binary-choice model can produce a predicted probability for each observation, not just a label such as “event” or “no event.” These are different outputs. A probability forecast describes uncertainty about the outcome; a classification rule turns that forecast into a category by applying a threshold. Evaluating the probabilities therefore requires more than counting how often classifications are correct.

Let $Y_i$ be the binary outcome for observation $i$, with $Y_i=1$ for an event and $Y_i=0$ otherwise, and let $p_i$ be the forecast probability assigned to that event. A forecast must satisfy $0\leq p_i\leq 1$. For example, $p_i=0.70$ means the model assigns a 70% probability to the event for observation $i$. In an evaluation data set, both $p_i$ and the subsequently observed $Y_i$ are available.

The usual goal is to evaluate predictions for cases drawn from the setting where the forecasts will be used. An average score on a finite evaluation sample describes performance on those cases; extending that conclusion to a broader target population requires the evaluation cases to be suitable for that target. Dependence between observations or a nonrepresentative evaluation sample can affect how well the results generalize.

Definition: A forecast is calibrated when, among cases assigned a given forecast probability $p$, the event occurs with probability $p$. In population notation, this is $E[Y\mid p]=p$. Since $Y$ is binary, $E[Y\mid p]$ is the event probability conditional on the forecast.

Calibration concerns whether forecast probabilities agree with event frequencies over cases. It does not by itself say whether the model gives useful separation between cases with higher and lower event risk, nor whether a particular threshold produces many correct classifications. In a finite sample, exact calibration at every possible forecast value is rarely assessable: there may be few or no repeated cases with exactly the same $p_i$.

Calibration Plots and Grouped Summaries

A calibration plot summarizes that comparison by grouping observations with similar forecasts. For each group, plot the group’s mean forecast probability on the horizontal axis and the observed event fraction on the vertical axis. The 45-degree line represents perfect calibration: a group with mean forecast $0.70$ would have an event fraction of $0.70$ if its forecasts were exactly calibrated at that level.

The grouping is a practical summary, not a definition of calibration. Different bin boundaries can give different plots, and broad bins can conceal patterns within a bin. Always report or show the number of observations in each group when feasible: an event fraction based on a few cases is much more variable than the same fraction based on many cases. A point off the diagonal is evidence of a discrepancy in that sample, not proof that the underlying forecasting process is miscalibrated.

Worked Example: Reading a Calibration Plot

Suppose a hypothetical evaluation data set contains eight cases. Four forecasts fall in a lower-probability group and four in a higher-probability group. Their forecasts and outcomes are:

Group	Forecast probabilities	Observed outcomes
Lower	0.20, 0.30, 0.40, 0.40	0, 0, 1, 0
Higher	0.60, 0.70, 0.80, 0.90	1, 1, 0, 1

For the lower group, the mean forecast is $(0.20+0.30+0.40+0.40)/4=0.325$. One of four outcomes is an event, so the observed event fraction is $1/4=0.25$. The plotted point is $(0.325,0.25)$, slightly below the diagonal: in this group, the observed event frequency is lower than the average forecast.

For the higher group, the mean forecast is $(0.60+0.70+0.80+0.90)/4=0.75$. Three of four outcomes are events, so the observed event fraction is $3/4=0.75$. Its plotted point is $(0.75,0.75)$, on the diagonal. This small example suggests reasonable agreement in the higher group and a modest discrepancy in the lower group, but four cases per group are far too few to establish reliable population calibration.

Proper Scores for Probability Forecasts

A score evaluates a forecast against its eventual outcome. The Brier score and log loss are two standard scores for binary probability forecasts. Both are losses, so smaller values indicate better performance. They assess the probabilities themselves, rather than first converting them to event/no-event labels.

Definition: For $n$ evaluated binary outcomes $Y_i$ and forecasts $p_i$, the mean Brier score is $$ \mathrm{BS}=\frac{1}{n}\sum_{i=1}^{n}(p_i-Y_i)^2. $$ The individual loss is between 0 and 1. A forecast of 1 for an observed event or 0 for an observed non-event has zero loss; a confident forecast in the wrong direction has a large loss.

Definition: The log loss for a binary forecast is $$ \mathrm{LL}=-\frac{1}{n}\sum_{i=1}^{n}\left[Y_i\ln(p_i)+(1-Y_i)\ln(1-p_i)\right]. $$ For an event, its individual loss is $-\ln(p_i)$; for a non-event, it is $-\ln(1-p_i)$. A forecast assigning probability zero to an outcome that occurs has infinite log loss.

The Brier score is bounded, while log loss is unbounded above and penalizes confident errors especially heavily. For meaningful comparisons, calculate scores on the same cases with the same outcome definition. Neither score has a universal cutoff that makes a model “good” in every application; the relevant scale and the purpose of the forecasts matter.

Worked Example: Calculating and Comparing Two Sets of Scores

Consider four hypothetical cases, all of which experience the event. Method A forecasts probabilities $0.80, 0.70, 0.60, 0.80$; Method B forecasts $0.90, 0.80, 0.50, 0.90$. Thus $Y_i=1$ for each case. These data are deliberately small and event-only; they illustrate arithmetic, not a general model comparison.

For Method A, the squared errors are $0.20^2, 0.30^2, 0.40^2,$ and $0.20^2$. Therefore,

\mathrm{BS}_A=\frac{0.04+0.09+0.16+0.04}{4} =\frac{0.33}{4}=0.0825.

Its total log loss is $-\ln(0.80)-\ln(0.70)-\ln(0.60)-\ln(0.80)$. Evaluating the logarithms before rounding gives a total of approximately $1.3138$, and the mean log loss is $1.31378766\ldots/4\approx 0.3284$.

For Method B, the squared errors are $0.10^2, 0.20^2, 0.50^2,$ and $0.10^2$, giving

\mathrm{BS}_B=\frac{0.01+0.04+0.25+0.01}{4} =\frac{0.31}{4}=0.0775.

Its total log loss is $-\ln(0.90)-\ln(0.80)-\ln(0.50)-\ln(0.90)\approx 1.1270$, so its mean is approximately $1.12701176/4=0.2818$. Both scores are lower for Method B on these four cases. That does not establish that it will perform better in a target population: all four outcomes are events, and a broader assessment should include the relevant range of cases and outcomes.

Why These Scores Reward Honest Probabilities

A scoring rule is called proper when, in expectation, a forecaster minimizes expected loss by reporting the true event probability. It is strictly proper when that minimum is unique. This property gives both scores a principled interpretation: they reward accurate probability statements, not merely successful classifications.

To verify this for the Brier score, suppose the true event probability for a case is $q$, and consider reporting $r$. The outcome is 1 with probability $q$ and 0 with probability $1-q$. The expected squared-error loss is

$$ q(1-r)^2+(1-q)r^2 =q-2qr+r^2 =q(1-q)+(r-q)^2. $$

The first term, $q(1-q)$, does not depend on the report $r$, and the second term is nonnegative, equaling zero only when $r=q$. Thus the expected Brier loss is uniquely minimized by reporting the true probability.

For log loss, the expected loss at report $r$, when $0<r<1$, is

L(r)=-q\ln(r)-(1-q)\ln(1-r).

For $0<q<1$, differentiating with respect to $r$ gives $L'(r)=-q/r+(1-q)/(1-r)$. Setting this equal to zero yields $(1-q)r=q(1-r)$, hence $r=q$. Also,

L''(r)=\frac{q}{r^2}+\frac{1-q}{(1-r)^2}>0,

so the stationary point is the unique minimum. If $q=0$ or $q=1$, the expected loss is minimized by reporting that same boundary probability; assigning zero probability to an outcome that then occurs incurs infinite loss. These results establish strict propriety for both scores, including the boundary cases.

Calibration Is Not Classification Accuracy

Classification accuracy depends on a threshold. For a chosen threshold $c$, classify a case as an event when $p_i\geq c$, and otherwise classify it as a non-event. Accuracy is the fraction of cases whose resulting labels match $Y_i$. Changing $c$ can change the classifications and accuracy while leaving every forecast probability—and therefore the Brier score, log loss, and calibration plot—unchanged.

Worked Example: The Threshold Changes Accuracy, Not the Forecasts

Use the eight forecasts and outcomes in the calibration example, with a threshold of $c=0.50$. All four lower-group cases are classified as non-events, and all four higher-group cases as events. Among the lower group, three outcomes are 0 and one is 1. Among the higher group, three are 1 and one is 0. The resulting counts are 3 true negatives, 3 true positives, 1 false negative, and 1 false positive.

Accuracy is $(3+3)/8=0.75$, or 75%. The classifier gets six of eight labels right. Yet this number does not reveal the calibration summaries: the lower group’s observed event fraction is 0.25 against a mean forecast of 0.325, while the higher group’s event fraction and mean forecast are both 0.75. Nor does accuracy distinguish the individual probability values within a group. It answers a threshold-specific label question, not whether the probabilities are well calibrated.

A model may classify many cases correctly but give probabilities that are systematically too high or too low. Conversely, its probabilities can be calibrated overall while a particular threshold gives an accuracy rate that is not useful for a decision. Which measure matters depends on the purpose: decisions based on a cutoff require attention to the threshold and the consequences of the two kinds of classification error, while probability-based decisions require credible probabilities and suitable scoring evaluation.

Common Pitfalls

Treating a good accuracy rate as evidence of calibrated probabilities. Accuracy discards the distance between a probability and the threshold; assess probability forecasts with calibration summaries and proper scores.
Calling a grouped calibration plot proof of calibration. A few points near the diagonal only describe the chosen bins and sample. Report bin sizes, consider the grouping, and interpret the plot as a diagnostic summary.
Comparing scores from different cases. Different outcome mixes or evaluation samples can make score differences hard to interpret. Compare forecasts on the same cases whenever possible.
Ignoring extreme log-loss penalties. A forecast of zero for an event, or one for a non-event, has infinite log loss. Check for boundary forecasts and understand any probability adjustments used before scoring.

Key takeaway: Calibration asks whether forecast probabilities match event frequencies; the Brier score and log loss measure forecast accuracy with proper scoring rules; classification accuracy measures threshold-based label agreement. These are related but distinct properties.

Check Your Understanding

Answer in terms of the forecast probabilities and observed outcomes.

A group has mean forecast probability 0.40 and an observed event fraction of 0.55. Where would its calibration-plot point lie relative to the 45-degree line?
For one case, the forecast is $p_i=0.25$ and the observed outcome is $Y_i=1$. Calculate its Brier loss and log loss.
Why can two models have identical classification accuracy at a given threshold but different Brier scores?
State the expected Brier loss when the true event probability is $q$ and the reported probability is $r$, and identify the minimizing report.
Why should a calibration plot show the number of observations in each probability group?