Calibrating Predicted Treatment Probabilities

Why Calibrate Treatment Probabilities?

Inverse probability of treatment weights use predicted treatment probabilities in their denominators. For a binary treatment, a small predicted probability for a treated person produces a large weight; a probability near one for a control does the same. This makes probability errors consequential: a prediction can look reasonable overall yet create unstable weights in the observations whose treatment was unlikely under the fitted model.

Calibration is one way to assess and, when appropriate, adjust the scale of predicted probabilities. It asks whether events occur at approximately the predicted rate among observations receiving similar predictions. It does not, by itself, establish that the treatment model has adequately accounted for confounding. The distinction matters: a recalibration can improve agreement between predictions and observed treatment frequencies while leaving important covariate patterns unrepresented.

For clarity, the un-stabilized inverse probability weight for a binary treatment is

Thus \(w_i=1/q_i\) for a treated individual and \(w_i=1/(1-q_i)\) for a control. Replacing \(q_i\) by a recalibrated value changes the denominator directly. All such calculations require probabilities strictly between zero and one. In a causal analysis, the usual identification assumptions, including consistency, no unmeasured confounding given the modeled covariates, and positivity, remain necessary. Calibration is not a substitute for them.

Calibration Methods and Their Meaning

A simple diagnostic groups predictions into bins and compares the average prediction in each bin with the observed treatment proportion. If a bin contains \(m\) individuals, its average prediction is \(\bar q\), and \(r\) of them received treatment, the observed proportion is \(r/m\). A difference between these values signals calibration error at that range of predictions. The binning is only a summary: results can depend on the number and boundaries of bins, and a small bin can have a noisy observed proportion.

A smooth alternative is logistic recalibration. Given original predictions \(q_i\), fit the calibration model

The fitted recalibrated probability is \(\hat q_i^{\,\mathrm{cal}}=\operatorname{expit}\{\hat\alpha+\hat\beta\operatorname{logit}(q_i)\}\), where \(\operatorname{expit}(x)=1/(1+\exp(-x))\). Ideal calibration in this model corresponds to \(\alpha=0\) and \(\beta=1\). Calibration-in-the-large fixes \(\beta=1\) and adjusts only the intercept; it shifts the predictions on the log-odds scale. A fitted slope below one pulls predictions toward the middle, while a slope above one spreads them farther apart, if the intercept is held at zero.

These procedures require observed treatment indicators paired with predictions, and the calibration sample must represent the setting where the predictions will be used. A recalibration estimated and evaluated on the same observations can appear better simply because it has adapted to their treatment outcomes. Independent validation data or cross-fitting can reduce this optimism. Calibration estimates can also be unstable when treatment outcomes are sparse or predictions lie near zero or one.

How Recalibration Changes Weights

The direction of weight changes can be determined without fitting a particular calibration model. Suppose an intercept adjustment adds \(\alpha\) to every logit, and write \(c=\exp(\alpha)>0\). Its probability mapping is

This mapping multiplies the odds by \(c\). The next result gives the corresponding changes in the two treatment-specific weights.

To verify the probability comparison, subtract \(q\) from the displayed expression for \(g_c(q)\):

The denominator and \(q(1-q)\) are positive, so the sign is the sign of \(c-1\). Since reciprocation reverses the ordering of positive numbers, \(g_c(q)>q\) implies \(1/g_c(q)<1/q\). It also implies \(1-g_c(q)<1-q\), and hence \(1/\{1-g_c(q)\}>1/(1-q)\). The reverse inequalities hold when \(c<1\). This establishes the result. \(\square\)

An intercept shift therefore does not simply “shrink all weights.” It lowers treated weights while raising control weights when it raises all probabilities. Whether the overall weight distribution becomes more or less dispersed depends on which treatment groups have which predictions and how much the shift changes them.

A slope adjustment can change weights differently across the prediction range. For instance, with \(\alpha=0\) and \(\beta=0.5\), logits are halved. A probability of \(0.25\) has odds \(1/3\); after the adjustment the odds are \(1/\sqrt{3}\), giving probability \(1/(1+\sqrt{3})\approx0.366\). A probability of \(0.75\) becomes \(1/(1+1/\sqrt{3})\approx0.634\). A treated person's weight at \(q=0.25\) therefore changes from \(4\) to approximately \(1/0.366=2.73\). A control's weight at \(q=0.75\) changes from \(4\) to approximately \(1/(1-0.634)=2.73\). Here the slope adjustment pulls both predictions toward \(0.5\) and reduces these example weights.

What Intercept Calibration Guarantees

Calibration-in-the-large has a precise in-sample property when its intercept is estimated by logistic likelihood with the original predictions as an offset. Let \(q_i\in(0,1)\) be fixed predictions and define \(q_i(\alpha)=\operatorname{expit}\{\operatorname{logit}(q_i)+\alpha\}\). Assume the sample contains at least one treated and one control individual. If a finite maximum-likelihood estimate \(\hat\alpha\) is used, then the average recalibrated prediction equals the observed treatment proportion.

For a complete derivation, the log-likelihood as a function of the intercept is

Since the derivative of \(\operatorname{logit}(q_i)+\alpha\) with respect to \(\alpha\) is one, differentiation gives

The likelihood is strictly concave. As \(\alpha\) tends to negative infinity, every \(q_i(\alpha)\) tends to zero, so \(\ell'(\alpha)\) tends to \(\sum_i A_i>0\). As \(\alpha\) tends to positive infinity, every \(q_i(\alpha)\) tends to one, so \(\ell'(\alpha)\) tends to \(\sum_i A_i-n<0\). Continuity gives a root, and strict concavity makes it unique. At the maximum, \(\ell'(\hat\alpha)=0\), so \(\sum_i A_i=\sum_i q_i(\hat\alpha)\). Dividing by \(n\) proves the equality. If all individuals have the same treatment value, there is no finite intercept maximum; the fitted intercept diverges toward a boundary. \(\square\)

This guarantee is deliberately limited. A single intercept can match the overall treatment rate while predictions are too low for one covariate group and too high for another. A calibration plot, subgroup checks, and weight diagnostics are therefore still needed. In particular, inspect extreme weights and the effective sample size, commonly summarized for weights \(w_i\) as \((\sum_i w_i)^2/\sum_i w_i^2\). A small value relative to \(n\) signals that a few observations dominate weighted summaries.

Choosing and Checking a Calibration Approach

Calibration is most useful when predictions are systematically too high or too low, or when their spread does not match observed treatment patterns. It is less persuasive when apparent errors are based on a few observations, when the calibration sample differs from the target population, or when a single summary hides substantial subgroup differences. Flexible binning or recalibration can itself overfit; predictions should be checked on data not used to fit the adjustment whenever possible.

A common mistake is to reason, “The recalibrated predictions match the observed treatment proportion, so the propensity model is correct and the weights remove confounding.” The correction is that calibration assesses agreement between predicted probabilities and treatment frequencies at the resolution being checked. Overall agreement does not guarantee agreement within covariate patterns relevant to treatment assignment, and neither agreement nor large weights can test the absence of unmeasured confounding. Compare calibration diagnostics with substantive knowledge of the treatment process, covariate coverage, and the resulting weight distribution.

Calibration is a post-processing option, not an automatic repair. Alternatives include revisiting the original treatment model, using a different probability estimator, or retaining the original predictions if the proposed recalibration is unsupported or unstable. Truncating extreme weights is a separate intervention: it directly modifies weights and can introduce bias, whereas calibration modifies predicted probabilities and then recomputes weights. Neither operation should be described as correcting all model misspecification.