Calibration Plots for Treatment Probability Models

What a Calibration Plot Checks

A treatment probability model assigns each person a predicted probability of receiving treatment, given the measured history available at the treatment decision. A calibration plot asks a direct question: among observations assigned similar predicted probabilities, did treatment occur about as often as predicted? This is a diagnostic of agreement between predicted and observed frequencies. It is not, by itself, a test that the model captures every feature of the treatment mechanism.

For a binary treatment \(A_i\), let \(\hat p_i\) be the fitted probability of treatment for observation \(i\). Divide the observations into groups \(G_1,\ldots,G_K\) according to their \(\hat p_i\) values. For group \(G_k\), let \(n_k\) be its size. Plot its mean predicted probability against its observed treatment frequency:

\bar p_k=\frac{1}{n_k}\sum_{i\in G_k}\hat p_i, \qquad \bar A_k=\frac{1}{n_k}\sum_{i\in G_k}A_i.

The horizontal coordinate is \(\bar p_k\), and the vertical coordinate is \(\bar A_k\). The reference line is the identity line \(y=x\). A point above that line means the group received treatment more frequently than its average prediction; a point below means treatment occurred less frequently than predicted. The vertical difference \(\bar A_k-\bar p_k\) is the group's calibration gap.

Definition: A grouped calibration plot compares, for each group formed using predicted treatment probabilities, the group's mean predicted probability \(\bar p_k\) with its observed treatment proportion \(\bar A_k\). The identity line represents agreement of these two quantities. The plot is a summary: agreement within groups does not establish that individual probabilities or the full conditional treatment model are correct.

Constructing and Reading the Plot

A practical construction begins by calculating fitted probabilities for all observations, choosing groups, and calculating \(\bar p_k\) and \(\bar A_k\) in each group. Quantile groups (such as deciles) tend to give groups of similar size; fixed-width groups keep probability ranges comparable but can leave some groups nearly empty. In either case, use the mean predicted probability in each group for the horizontal coordinate, not automatically the midpoint of the interval: predictions within a group need not be uniformly distributed.

The choice of number and boundaries of groups affects what is visible. Many narrow groups can produce a jagged plot with substantial sampling noise, while a few broad groups can hide local departures. Report group sizes, especially where treatment is rare, and do not treat small vertical deviations as meaningful without considering uncertainty. A smooth calibration curve can supplement grouped points, but its appearance depends on the smoothing method and bandwidth; it does not remove the need to examine the data support.

Worked Example: Reading Three Probability Groups

Suppose 60 independent observations are divided into three groups of 20. Their mean predicted treatment probabilities are 0.20, 0.50, and 0.80. The observed treatment counts are 3, 10, and 15, respectively. Calculate the plotted points and interpret their gaps.

For the first group, \(\bar A_1=3/20=0.15\), so the point is \((0.20,0.15)\), with gap \(0.15-0.20=-0.05\). For the second group, \(\bar A_2=10/20=0.50\), giving \((0.50,0.50)\), with gap \(0\). For the third group, \(\bar A_3=15/20=0.75\), giving \((0.80,0.75)\), with gap \(0.75-0.80=-0.05\).

The first and third groups have observed frequencies five percentage points below their predictions; the middle group agrees exactly. This pattern is a descriptive signal to investigate, not proof of model failure. In particular, the counts in the groups are finite, and the plot alone does not establish whether either five-point gap is larger than expected from sampling variation.

Why Correct Probabilities Calibrate on Average

The basic justification is an expectation identity. Let \(H_i\) denote the measured history available before treatment, and write the true treatment probability as \(p_0(H_i)=P(A_i=1\mid H_i)\). Suppose fitted probabilities are evaluated on observations not used to fit the model, or otherwise can be treated as fixed for the calculation below. If \(\hat p_i=p_0(H_i)\), then within any group selected using those fixed predictions, the expected observed frequency equals the mean predicted probability.

Result: Conditional on the histories and fixed predictions, suppose \(A_i\) has conditional mean \(p_0(H_i)\) and \(\hat p_i=p_0(H_i)\) for every observation. For any nonempty group \(G_k\) determined by those predictions, \(E[\bar A_k\mid H_1,\ldots,H_n]=\bar p_k\). Thus the identity line describes the conditional expectation of the plotted observed frequency under correct probabilities; an individual finite-sample point need not lie on that line.

To prove the result, condition on the histories and predictions. Group membership and \(n_k\) are then fixed. Since \(A_i\) is binary, \(E[A_i\mid H_1,\ldots,H_n]=p_0(H_i)=\hat p_i\). Linearity of expectation gives

E[\bar A_k\mid H_1,\ldots,H_n] =\frac{1}{n_k}\sum_{i\in G_k}E[A_i\mid H_1,\ldots,H_n] =\frac{1}{n_k}\sum_{i\in G_k}\hat p_i =\bar p_k.

This is a statement about an expectation, not a guarantee of exact equality in a realized data set. It also clarifies a limitation: a grouped plot checks calibration at the resolution of its grouping. Different individual probabilities can be wrong while their errors average out within a group.

Sampling Variation Is Not Always Binomial

Uncertainty around a plotted point matters because treatment indicators within a group need not have the same probability. Conditional on fixed histories, if treatment indicators are independent with probabilities \(p_i=p_0(H_i)\), the group count is generally a sum of independent Bernoulli variables with different probabilities—a Poisson-binomial sum. Its variance is not generally the binomial variance formed from the group's mean probability.

Result: For independent binary outcomes in a fixed group with probabilities \(p_i=P(A_i=1\mid H_i)\), the conditional variance of the observed group proportion is \(\operatorname{Var}(\bar A_k\mid H_1,\ldots,H_n)=\sum_{i\in G_k}p_i(1-p_i)/n_k^2\). If the outcomes are instead iid Bernoulli with a common probability \(p\), the group count is binomial and the variance is \(p(1-p)/n_k\). These are different assumptions.

For the first formula, independence makes the covariance terms zero. Since a Bernoulli variable with probability \(p_i\) has variance \(p_i(1-p_i)\), scaling the sum by \(1/n_k\) gives

\operatorname{Var}(\bar A_k\mid H_1,\ldots,H_n) =\frac{1}{n_k^2}\operatorname{Var}\left(\sum_{i\in G_k}A_i\mid H_1,\ldots,H_n\right) =\frac{1}{n_k^2}\sum_{i\in G_k}p_i(1-p_i).

When all probabilities equal \(p\), this reduces to \(p(1-p)/n_k\). In applications, dependence from clustering or repeated observations also changes the variance. Model fitting on the same observations can add dependence between predictions, group membership, and outcomes. Consequently, a simple binomial interval should not be presented as universally valid; uncertainty procedures should reflect the sampling design, probability heterogeneity, and fitting procedure as appropriate.

Worked Example: Heterogeneous Probabilities Change the Variance

Consider two independent observations with conditional treatment probabilities 0.10 and 0.90, placed in the same group. The observed group proportion is \(\bar A=(A_1+A_2)/2\). Calculate its variance and compare it with the binomial variance one might incorrectly use from the average probability.

The correct conditional variance is \(\{0.10(0.90)+0.90(0.10)\}/2^2=(0.09+0.09)/4=0.045\). The average probability is \((0.10+0.90)/2=0.50\). Treating the two outcomes as iid Bernoulli with probability 0.50 would give variance \(0.50(0.50)/2=0.125\), which is not the conditional variance in this example.

The difference can also be checked from the distribution of the count \(A_1+A_2\): its probabilities for 0, 1, and 2 treated observations are \(0.10(0.90)=0.09\), \(0.10(0.10)+0.90(0.90)=0.82\), and \(0.90(0.10)=0.09\). Thus \(\bar A\) takes values 0, 0.5, and 1 with those probabilities. Its mean is \(0(0.09)+0.5(0.82)+1(0.09)=0.50\); its variance is \(0.09(0.50^2)+0.82(0^2)+0.09(0.50^2)=0.045\), as calculated above.

Longitudinal Treatment Decisions

For longitudinal treatment, define the target probability at time \(t\) as \(P(A_{it}=1\mid H_{it})\), where \(H_{it}\) is the observed history before that decision. A useful plot groups fitted probabilities for a given decision time and compares the group's observed treatment frequency at that time with its average prediction. One may also present time-specific panels to see whether calibration differs over follow-up.

Pooling decision times can conceal opposing departures. Time-specific treatment mechanisms and histories may differ, so equality after pooling is not evidence of agreement at each decision. Conversely, time-specific groups may be small; the plot should show their sizes and be interpreted cautiously. Repeated records from one person are dependent, so uncertainty calculations must account for that dependence rather than treating all person-times as independent.

Worked Example: Pooling Can Hide Time-Specific Gaps

At time 1, 100 decisions have mean predicted probability 0.20 and 30 observed treatments. At time 2, 100 decisions have mean predicted probability 0.80 and 70 observed treatments. Compare the time-specific plots with a pooled calculation.

At time 1, the observed frequency is \(30/100=0.30\), so the point is \((0.20,0.30)\) and the gap is \(+0.10\). At time 2, it is \(70/100=0.70\), giving \((0.80,0.70)\) and a gap of \(-0.10\).

After pooling, the mean predicted probability is \((100(0.20)+100(0.80))/200=0.50\), and the observed frequency is \((30+70)/200=0.50\). The pooled point is \((0.50,0.50)\), exactly on the identity line. It hides the two opposite time-specific gaps. The example shows why a pooled calibration summary should be supplemented by plots stratified by decision time when treatment mechanisms vary over follow-up.

How Calibration Fits With Other Diagnostics

Calibration is one part of treatment-model assessment. It asks whether predicted probabilities agree with observed frequencies in groups. Covariate balance after weighting asks a different question: whether the weighted treatment groups are comparable on measured covariates. As emphasized in the earlier material on why propensity model fit is not balance, neither a good calibration plot nor a good-looking model-fit summary guarantees balance. Conversely, balance is not a direct check that predicted treatment probabilities match observed frequencies.

Calibration also differs from discrimination, which concerns how well predictions separate treated from untreated observations; the standard name for that alternative is discrimination. Neither summary substitutes for causal assumptions such as consistency, exchangeability given measured history, and positivity. A plot can reveal an empirical mismatch, but it cannot show that all confounders were measured or that a causal effect is identified.

Several warning signs merit follow-up: sustained points on one side of the identity line, departures concentrated at high or low predicted probabilities, unstable patterns across time, and groups with few observations or very few treated outcomes. These signals can arise from misspecification, but also from sampling variation, sparse support, data problems, or model fitting. Calibration plots indicate where to investigate; they do not determine the cause of a departure.

Common Pitfalls and Limitations

Using a binomial standard error without its assumptions. A binomial variance requires a common success probability and independent Bernoulli outcomes, or a justified iid marginal sampling setup. With fixed, heterogeneous probabilities, use the sum of \(p_i(1-p_i)\) divided by \(n_k^2\); account for clustering and fitting where relevant.
Equating grouped calibration with a correct conditional model. Errors in individual predictions can cancel within a bin. Finer groups may reveal more detail, but they also contain less information.
Ignoring how groups were formed. Sparse or empty intervals make points noisy or undefined. Show group sizes and state the grouping rule.
Relying only on an in-sample plot. Fitted values and outcomes from the same data can make apparent agreement optimistic. Evaluation on held-out observations or a resampling procedure can help assess this dependence, though each approach has its own uncertainty.
Interpreting the identity line as a causal guarantee. Calibration concerns predicted frequencies, not unmeasured confounding, positivity, or the validity of the causal design.

The main practical alternative is not a single competing plot but a complementary diagnostic set: inspect calibration, assess weighted covariate balance, examine support and extreme weights, and compare results across defensible modeling choices. Calibration is especially useful for locating systematic over- or underprediction, while those other checks address different failure modes. None alone certifies a treatment model.

Check Your Understanding

Answer each question, then check the brief solutions below.

A group has 40 observations, mean predicted probability 0.35, and 18 observed treatments. What point is plotted, and what is its calibration gap?
Why is the identity line a conditional expectation benchmark rather than a requirement that every observed point lie exactly on the line?
Three independent observations have fixed treatment probabilities 0.20, 0.50, and 0.80. Find the variance of their observed treatment proportion.
Give one reason to plot longitudinal treatment decisions separately by time.
Can perfect grouped calibration establish covariate balance or absence of unmeasured confounding? Explain.

Solutions. (1) The observed proportion is \(18/40=0.45\), so plot \((0.35,0.45)\); the gap is \(0.45-0.35=0.10\). (2) Under correct probabilities the expected group frequency equals the mean prediction, but random outcomes fluctuate around that expectation in a finite sample. (3) The variance is \(\{0.20(0.80)+0.50(0.50)+0.80(0.20)\}/3^2=(0.16+0.25+0.16)/9=0.57/9\approx0.0633\). (4) Opposing departures at different times can cancel in a pooled plot, as in the worked example. (5) No. Calibration summarizes observed-versus-predicted frequencies in groups; balance and unmeasured confounding are distinct issues.