Calibration of Estimated Propensity Scores

What Calibration Says About Estimated Propensity Scores

An estimated propensity score is a predicted probability of receiving treatment given measured covariates. Calibration asks whether those probabilities agree with observed treatment frequencies. If a collection of individuals is assigned scores near 0.30, for example, a calibrated model should predict that about 30% of such individuals receive treatment, on average. This is a statement about groups of predictions, not a claim that each person's probability can be verified from one observed treatment assignment.

Calibration is useful for assessing whether estimated scores behave like probabilities. It is not, by itself, evidence that the covariates needed for a causal analysis were included, that treatment groups are comparable, or that a causal effect estimate is unbiased. In particular, calibration and covariate balance answer different questions.

The exact population definition is an ideal: there may be few or no individuals with precisely the same score. A grouped diagnostic approximates it by dividing scores into ranges. For group \(g\), let \(n_g\) be its number of individuals. Its mean predicted probability and observed treatment rate are

A group is well calibrated when \(\bar{\hat e}_g\) and \(\hat p_g\) are close. A positive difference \(\hat p_g-\bar{\hat e}_g\) means the group had a higher treatment rate than the model predicted; a negative difference means the predicted rate was higher than the observed rate. Because each group contains a finite number of individuals, the two quantities need not match exactly even when predictions are well calibrated.

Grouped Calibration in Practice

A calibration table reports, for each score group, its size, mean estimated score, observed treatment rate, and their difference. Groups may be defined using prespecified score intervals or by dividing the sample into quantile groups. Quantile groups aim for similar group sizes, while fixed intervals retain the same score ranges across samples. Either choice is a summary: changing group boundaries can change the appearance of the diagnostic.

One summary of grouped discrepancies is the weighted absolute calibration error:

This quantity is the sample-size-weighted average absolute gap between observed and predicted rates across the chosen groups. It is expressed in probability units, so an ECE of 0.04 is an average gap of four percentage points under that grouping. It does not have a universal pass-or-fail threshold; its value depends on the grouping and the sample.

An overall comparison is the difference between the average predicted probability and the observed treatment proportion:

This is often called a calibration-in-the-large discrepancy. A value near zero means the model's total predicted number of treated individuals is close to the observed number. It does not show whether predictions agree with treatment frequencies across the range of scores. Positive and negative group discrepancies can cancel in the overall difference.

Why Logistic Maximum Likelihood Can Look Calibrated Overall

For ordinary, unpenalized logistic regression with an intercept, there is a useful algebraic identity. It concerns the fitted sample as a whole and should not be mistaken for a guarantee of calibration within score groups. To derive it correctly, first express the likelihood in terms of parameter-dependent probabilities. Write \(x_i\) for the covariate vector, including an intercept, and define \(e_i(\beta)=\operatorname{logit}^{-1}(x_i^T\beta)\). For observed treatment indicators \(A_i\), the log-likelihood is

The derivative with respect to the intercept coefficient \(\beta_0\) is obtained while \(e_i(\beta)\) still depends on \(\beta_0\). Since \(\partial e_i(\beta)/\partial\beta_0=e_i(\beta)[1-e_i(\beta)]\), differentiating each likelihood contribution gives

If a finite, unpenalized maximum-likelihood estimate \(\hat\beta\) exists, the likelihood score is zero at that estimate. Substituting \(\hat e_i=e_i(\hat\beta)\) only after taking the derivative gives

The fitted probabilities therefore sum to the observed number treated, and their overall mean equals the observed treatment proportion. This identity relies on the specified likelihood, an intercept, and a finite maximum-likelihood estimate. It is not a general property of every estimation method; for example, a penalty can change the likelihood score equation.

Most importantly, the identity constrains only the total. For instance, suppose two equally sized score groups have mean predictions 0.20 and 0.60, while their observed treatment rates are 0.30 and 0.50. The average prediction is \((0.20+0.60)/2=0.40\), and the overall observed rate is also \((0.30+0.50)/2=0.40\). Yet the group discrepancies are \(0.30-0.20=0.10\) and \(0.50-0.60=-0.10\). The aggregate agreement conceals ten-percentage-point errors in opposite directions. This arithmetic illustrates why grouped checks remain useful even when the overall totals agree.

Diagnostics, Comparisons, and Limitations

A grouped table or calibration plot makes patterns easier to see than one aggregate number. Plot mean predicted probability on the horizontal axis and observed treatment rate on the vertical axis, with the 45-degree line representing agreement. Points consistently above that line indicate underprediction of treatment frequency in those groups; points below indicate overprediction. Also inspect group sizes and treatment counts: a rate from a small group is inherently more variable than one from a large group.

Bins simplify interpretation but discard within-group detail. A wide bin can average together regions where the model overpredicts and underpredicts, making the bin appear well calibrated. Very narrow bins can produce unstable rates or empty groups. Fixed score intervals and quantile groups offer different trade-offs, so report how groups were defined and avoid treating one grouping as definitive. A smoother calibration curve can reveal patterns between bins, but it too is an estimate and can vary with the smoothing choices.

Calibration is different from treatment discrimination. Discrimination concerns whether individuals who receive treatment tend to have higher predicted scores than those who do not; calibration concerns whether predicted probabilities match observed frequencies. A model may rank individuals well but systematically predict probabilities that are too high or too low. Conversely, a model predicting the overall treatment proportion for everyone can be calibrated on average while providing little ranking information.

Calibration is also not the same as covariate balance after using propensity scores. Calibration compares predicted treatment probabilities with treatment outcomes. Balance diagnostics assess whether measured covariates are comparable across treatment groups after a specified adjustment. A calibrated treatment-prediction model does not by itself show that the covariates needed for the causal question were measured and modeled adequately. The introductory material on the balancing property gives the separate role of that property; calibration should not be used as its substitute.

Finally, grouped calibration is a sample diagnostic, not proof that estimated probabilities equal the true propensity scores for every individual. A good grouped fit can coexist with poor calibration inside groups, and a poor fit can reflect finite-sample variation as well as model misspecification. Interpret discrepancies alongside group sizes, the prediction range, and the purpose for which the scores will be used.

Common Pitfall

• Wrong reasoning: “The average fitted score equals the observed treatment proportion, so the propensity model is calibrated.”
• Correction: Overall equality checks only one aggregate constraint. Group-specific errors can cancel, as in the two-group calculation above. Compare observed rates with mean predictions over multiple score ranges, and remember that even good calibration does not establish covariate balance or causal validity.
• Wrong reasoning: “A group’s observed rate must equal its predicted rate exactly if the model is correct.”
• Correction: Treatment is a binary outcome, so observed rates vary from sample to sample. A difference is a diagnostic signal to interpret in light of group size and sampling variability, not automatic proof of failure.