Calibration-in-the-Large for Binary Predictions

Why Compare Observed and Predicted Risk?

For binary outcomes, a prediction \(p_i\) is an individual's predicted probability of an event, while the observed outcome \(Y_i\) is either 0 or 1. A model can give risks that are systematically too high or too low even when its predictions distinguish higher-risk from lower-risk individuals. A direct first check is therefore to compare the average predicted risk with the observed event frequency.

That aggregate comparison has a closely related model-based summary: the calibration-in-the-large intercept, estimated by adding an intercept to the predicted log-odds while holding their coefficient fixed at one. The raw comparison and the intercept answer related, but not identical, questions. Keeping them distinct avoids a common interpretive error.

Two Summaries of Calibration-in-the-Large

Definition: Suppose \(Y_1,\ldots,Y_n\) are observed binary outcomes, \(Y_i\in\{0,1\}\), and \(p_1,\ldots,p_n\) are their predicted event probabilities. Define the observed event count \(O=\sum_{i=1}^n Y_i\), the total predicted event count \(E=\sum_{i=1}^n p_i\), and the mean predicted risk \(\bar p=E/n\). The observed event frequency is \(\bar Y=O/n\). The raw count discrepancy is \(D=O-E\), and the raw frequency discrepancy is \(\bar Y-\bar p=D/n\). A positive discrepancy means observed frequency exceeds mean predicted risk; a negative discrepancy means mean predicted risk exceeds observed frequency.

For example, \(E=12\) means the predictions sum to 12 expected events across the \(n\) individuals. It does not mean that exactly 12 events must occur. The observed count \(O\) is an integer; \(E\) need not be.

When \(E>0\), the ratio \(O/E\) is another way to express the comparison: it is above one when events are more frequent than predicted in aggregate, below one when they are less frequent, and equal to one when \(O=E\). When \(E=0\), this ratio is undefined, so the difference \(D\) is the safer summary.

Definition: For fixed predictions satisfying \(0<p_i<1\), consider the offset logistic model \[ \operatorname{logit}\{q_i(a)\}=\operatorname{logit}(p_i)+a, \qquad \operatorname{logit}(u)=\log\!\left(\frac{u}{1-u}\right), \] where \(q_i(a)\) is the event probability under the recalibrated model and \(a\) is a common intercept adjustment. When a finite maximum-likelihood estimate exists, the calibration-in-the-large intercept \(\hat a\) is that estimate. A positive \(\hat a\) shifts all predicted log-odds upward; a negative \(\hat a\) shifts them downward.

The offset is the original predicted log-odds, not an estimated coefficient. Thus this model estimates only a common shift. It does not fit a new slope or assess how calibration varies across the risk range.

Worked Examples: Reading the Aggregate Comparison

Worked Example: Systematic Underprediction

Four individuals have predicted risks \(0.2,0.2,0.2,0.2\), and their observed outcomes are \(1,0,1,0\). The observed count is \(O=1+0+1+0=2\). The total predicted count is \(E=4(0.2)=0.8\). Therefore \(\bar Y=2/4=0.5\), \(\bar p=0.8/4=0.2\), and \(D=2-0.8=1.2\). The predictions understate the event frequency by \(0.5-0.2=0.3\), or 30 percentage points.

Here the offset-model estimate can also be found exactly. The adjusted probability is the same for all four individuals. To match the observed frequency, it must equal \(2/4=0.5\). Since \(\operatorname{logit}(0.5)=0\) and \(\operatorname{logit}(0.2)=\log(1/4)\), \[ \hat a=0-\log(1/4)=\log(4)\approx1.386. \] Indeed, adding \(\log(4)\) to the original log-odds gives zero, whose inverse-logit is \(0.5\). The positive intercept agrees with the raw evidence of underprediction.

Worked Example: Systematic Overprediction

Suppose five individuals each have predicted risk \(0.6\), and exactly one experiences the event. Then \(O=1\), \(E=5(0.6)=3\), \(\bar Y=1/5=0.2\), and \(\bar p=3/5=0.6\). Thus \(D=1-3=-2\), and the mean predicted risk exceeds the observed frequency by \(0.6-0.2=0.4\), or 40 percentage points.

The offset-model probability must be \(0.2\) for each individual to match the observed frequency. Consequently, \[ \hat a=\operatorname{logit}(0.2)-\operatorname{logit}(0.6) =\log(1/4)-\log(3/2) =\log(1/6)\approx-1.792. \] The corresponding odds are multiplied by \(e^{\hat a}=1/6\). The negative adjustment lowers the predicted risks, as expected when the original risks were too high overall.

The Offset Intercept: Score Equation and Existence

The offset-model intercept is not generally obtained by taking the logit of the observed frequency and subtracting the logit of the mean predicted risk. When predictions differ across individuals, each probability changes nonlinearly as the same intercept is added. Instead, its estimate is characterized by the model's likelihood equation.

Theorem: Suppose all \(p_i\) are fixed and strictly between 0 and 1. The offset logistic log-likelihood has a finite, unique maximizer if the observed outcomes include at least one 0 and at least one 1. In that case, \(\hat a\) is the unique solution of \[ \sum_{i=1}^n q_i(a)=O. \] If all outcomes are zero, there is no finite maximizing intercept; the likelihood supremum is approached as \(a\to-\infty\). If all outcomes are one, there is no finite maximizing intercept; the likelihood supremum is approached as \(a\to+\infty\).

Proof. Write \(\eta_i=\operatorname{logit}(p_i)\), so \(q_i(a)=e^{\eta_i+a}/\{1+e^{\eta_i+a}\}\). Up to terms that do not depend on \(a\), the log-likelihood is \[ \ell(a)=\sum_{i=1}^n\left[Y_i(\eta_i+a)-\log\{1+e^{\eta_i+a}\}\right]. \] Differentiating gives \[ \ell'(a)=O-\sum_{i=1}^n q_i(a), \qquad \ell''(a)=-\sum_{i=1}^n q_i(a)\{1-q_i(a)\}<0 \] for every finite \(a\). Thus the log-likelihood is strictly concave, and any finite solution of the score equation is the unique maximizer.

If the outcomes include both values, then \(0<O<n\). As \(a\to-\infty\), every \(q_i(a)\to0\), so \(\ell'(a)\to O>0\). As \(a\to+\infty\), every \(q_i(a)\to1\), so \(\ell'(a)\to O-n<0\). The derivative is continuous, so it has a zero at a finite value of \(a\). Strict concavity makes that zero unique.

If all outcomes are zero, then \(\ell'(a)=-\sum_iq_i(a)<0\) for every finite \(a\); the likelihood increases as \(a\) decreases and approaches its supremum as \(a\to-\infty\). If all outcomes are one, then \(\ell'(a)=n-\sum_iq_i(a)>0\) for every finite \(a\); the likelihood increases as \(a\) increases and approaches its supremum as \(a\to+\infty\). Neither boundary case has a finite maximizing intercept. \(\square\)

Key consequence: Under the theorem's conditions, \(\hat a=0\) if and only if \(O=E\). In fact, the score at zero is \(\ell'(0)=O-\sum_i p_i=O-E\), and strict concavity makes its zero the unique maximizer. This equivalence concerns the offset intercept and the raw count equality; it does not mean that the intercept equals a simple transformation of the frequency discrepancy.

Worked Example: Heterogeneous Risks and the Score Equation

Consider three predictions \(p_1=0.1\), \(p_2=0.2\), and \(p_3=0.7\), with outcomes \(Y_1=1\), \(Y_2=0\), and \(Y_3=1\). Then \(O=2\), \(E=0.1+0.2+0.7=1\), \(\bar Y=2/3\), and \(\bar p=1/3\). The raw discrepancy is \(D=1\), so the predictions understate the observed frequency by \(1/3\).

To find the offset intercept, set \(c=e^a\). The adjusted probability for an original risk \(p_i\) is \[ q_i(a)=\frac{c p_i}{1-p_i+c p_i}. \] The score equation requires \[ \frac{0.1c}{0.9+0.1c} +\frac{0.2c}{0.8+0.2c} +\frac{0.7c}{0.3+0.7c}=2. \] For \(c\approx6.79\), the three terms are approximately \(0.4300\), \(0.6293\), and \(0.9406\), respectively; their sum is approximately \(1.9999\), with the small discrepancy due to rounding. Hence \(\hat a\approx\log(6.79)\approx1.915\). The estimate is positive, but the adjusted probabilities are not all equal: each responds to the same shift in log-odds from its own starting risk.

Boundary Cases and Common Mistakes

Worked Example: Zero Predicted Events and No Finite Intercept

Suppose \(n=2\), \(p_1=p_2=0\), and \(Y_1=Y_2=0\). Then \(O=0\), \(E=0\), and \(D=O-E=0\). The mean predicted risk and observed frequency are both zero, so the raw frequency difference is zero. But \(O/E=0/0\) is undefined, not one.

Moreover, the offset logistic model above requires \(0<p_i<1\), because \(\operatorname{logit}(p_i)\) is not finite at \(p_i=0\). This example therefore does not have a finite offset-model intercept as defined here. It illustrates why the raw comparison and the offset model have different conditions of use.

Even with predictions strictly between zero and one, a finite offset intercept need not exist. For example, if \(p_1=0.1\), \(p_2=0.2\), and both outcomes are zero, then \(O=0\) while \(E=0.3\). The raw difference indicates overprediction, but the likelihood has no finite maximizing intercept: its supremum is approached as \(a\to-\infty\). A statement that the estimate is always finite would be incorrect.

Confusing counts with frequencies. \(D=O-E\) is a count difference; \(\bar Y-\bar p=D/n\) is a difference in proportions. State which scale is being reported.
Using \(O/E\) when \(E=0\). The ratio is undefined in that case. Report \(D\) or the two frequencies instead.
Calling the raw comparison the fitted intercept. The raw difference is directly \(O-E\); the offset intercept solves \(\sum_iq_i(a)=O\). For heterogeneous risks, it requires solving that equation.
Assuming every intercept estimate is finite. With all-zero or all-one outcomes and strictly interior predictions, the likelihood has no finite maximizing intercept.

Calibration-in-the-large is useful because it reveals a systematic aggregate shift: do the predictions, on average, assign too much or too little risk? It is not a complete check of calibration. Two sets of predictions can have the same mean while differing in how well their risks correspond to outcomes at different risk levels. An aggregate match should therefore not be interpreted as proof that every part of the prediction scale is well calibrated.

Check Your Understanding

Use the definitions and score equation to answer the following.

For six individuals, \(O=3\) and \(E=2.4\). Find the raw count discrepancy and the frequency discrepancy. What direction of aggregate error do they indicate?
If \(E>0\), what does \(O/E=1\) imply about the raw discrepancy? Why can this ratio not be used when \(E=0\)?
State the offset-model score equation for the calibration-in-the-large intercept and explain why it is not generally a logit of the observed frequency minus a logit of the mean predicted risk.
Under what outcome condition does the offset model have a finite, unique maximum-likelihood intercept when all predictions are strictly between zero and one?
Why does a zero raw discrepancy not establish that predicted risks are accurate at every risk level?