Worked Comparison of Flexible Propensity Estimators

Why Compare Flexible Propensity Estimators?

A propensity score is the probability of receiving treatment conditional on observed covariates. Different model classes can estimate that probability in different ways: a generalized additive model can represent smooth, nonlinear covariate effects; a random forest can represent interactions through collections of trees; and gradient boosting can build a prediction function by adding trees in sequence. Their fitted probabilities need not agree, even when the models use the same data and covariates.

The practical question is not simply which method predicts treatment most accurately. Propensity scores are used in analyses that seek to adjust for observed covariates, so useful checks include whether probabilities are sensible, whether predictions are calibrated, and whether the estimated scores support adequate covariate balance in the intended analysis. Calibration is informative, but it is not a direct test of balance or of the assumptions needed for a causal interpretation.

The comparison below focuses on estimating these probabilities. For a causal interpretation, the analysis also needs substantive conditions such as consistency, conditional exchangeability given the measured covariates, and positivity. Estimating a score does not establish those conditions; in particular, no comparison of prediction algorithms can rule out unmeasured confounding.

Three Models, One Prediction Target

All three methods aim to estimate \(P(T=1\mid X)\), but impose different structures on how the covariates relate to treatment. In a logistic generalized additive model (GAM), a typical specification is

Here the \(f_j\) are smooth functions estimated from the data. This can capture nonlinear effects without requiring each effect to be a straight line on the log-odds scale. In this common additive form, however, covariate effects do not automatically interact; interactions need to be included if warranted. Smoothing choices also matter: a function that is too constrained can miss structure, while one that is too flexible can follow noise.

A random forest fits many trees using resampled data and, typically, randomly selected candidate predictors at tree splits. Its probability prediction aggregates the trees’ treatment-probability predictions. This approach can capture nonlinearities and interactions, but the result depends on choices such as tree depth, the number of candidate predictors, and the number of trees. Averaging can produce stable predictions, but does not guarantee good calibration.

Gradient boosting builds an additive sequence of trees, with later trees chosen to improve the current predictions under a specified loss function. For binary treatment, logistic loss is a common choice. The learning rate, tree depth, and number of boosting steps regulate how complex the fitted function becomes. Too little boosting can underfit; too much can fit idiosyncrasies in the training sample.

These brief descriptions identify the models being compared, not a universal specification for each. Different implementations and tuning choices can produce different predictions. For a fair comparison, use the same units, treatment definition, candidate covariates, and evaluation rows, and document the modeling and tuning choices. A single selected fit from each model class is a comparison of those fits, not proof that one entire model class is superior.

A Hypothetical Dataset and Three Sets of Predictions

Consider 12 hypothetical units. Before examining treatment, an analyst groups them into three prespecified covariate profiles: low, middle, and high treatment propensity based on observed covariate patterns. Each profile contains four units. One, two, and three units, respectively, received treatment. The treatment rates are therefore 0.25, 0.50, and 0.75. These profiles are a compact way to display this example; they are not a substitute for checking individual-level covariate balance.

Suppose a GAM, random forest, and gradient-boosting model are fit using the same candidate covariates and a common tuning protocol. To avoid assessing each model on the very observations used to fit it, suppose the displayed values are cross-fitted predictions: each unit’s probability was predicted by a model fit without that unit. The table reports the average prediction within each profile. The numerical outputs are hypothetical.

Calibration: What the Grouped Comparison Shows

A model is calibrated in a group, in a practical empirical sense, when the mean predicted probability in that group is close to the observed treatment rate. In a population statement, calibration means that among units assigned a predicted probability \(q\), the treatment probability is \(q\). With a finite dataset, this relationship is assessed approximately, often by grouping predictions and comparing group means. Results depend on how groups are formed, and sparse groups can make the comparison noisy.

For this example, compare each model’s mean prediction with the observed rate in each of the three prespecified profiles. A weighted mean absolute calibration gap is

where \(n_g\) is the number of units in profile \(g\), \(\bar{\hat e}_g\) is the mean predicted probability there, and \(\hat p_g\) is the observed treatment rate there. This is a descriptive summary for these particular groups, not a universal measure of calibration.

Comparing Predictions Without Declaring a Winner

A useful comparison asks both how well predictions align with observed treatment frequencies and how much the estimates differ across methods. In the table, the three methods order the profiles in the same way, from lower to higher estimated probability. But their levels differ. For the high profile, the random forest predicts 0.82 on average, compared with 0.72 for the GAM, a difference of

That is a 10-percentage-point difference in average predicted treatment probability for the profile. It may reflect different assumptions about the covariate-response relationship, tuning choices, or sampling variability. It does not, by itself, show which estimate is correct. Nor does the close match between a group’s predicted and observed treatment rates guarantee that the resulting propensity scores balance every relevant covariate.

A Short Justification for Cross-Fitted Calibration Checks

When predictions are evaluated on the same observations used to fit a flexible model, the model has already used those treatment labels to choose its predictions. Its apparent agreement with the observed treatment rates can therefore be overly optimistic. Cross-fitting separates prediction from the treatment label of the unit being predicted: partition the data into folds, fit the model on the other folds, and predict the held-out fold. Repeating this yields an out-of-fold prediction for each unit.

The logic is that the held-out unit’s treatment label did not directly determine its own fitted prediction. Consequently, the resulting grouped comparison is a more honest check of prediction performance than an in-sample comparison, although it can still be noisy and depends on the fold construction and tuning procedure. If model tuning itself uses treatment outcomes, that tuning must also be handled within the training data for each fold to avoid information leakage.

Diagnostics, Alternatives, and Limitations

Calibration is one diagnostic among several. Check whether predictions are concentrated near zero or one, whether estimates are implausibly constant, and whether calibration changes across relevant covariate subgroups. Extreme estimated scores can indicate limited overlap or model behavior that warrants investigation; a well-calibrated overall average does not make such concerns disappear. The earlier material on calibration, cross-validation, out-of-fold prediction, and extreme fitted scores gives the relevant diagnostic foundations.

The main alternatives in this comparison make different compromises. A specified logistic model can be easier to interpret but may miss nonlinearities or interactions if its specification is too restrictive. A GAM can allow smooth effects while retaining an additive structure, but does not automatically capture all interactions. Random forests and boosting can represent more complex patterns, but their predictions depend on tuning and may be less transparent. These are tendencies, not guarantees: a simple model can predict well when its structure is appropriate, and a flexible model can perform poorly.

A fair decision should consider the purpose of the score, not only classification or calibration summaries. For propensity-score use, evaluate balance on the covariates relevant to the analysis after applying the intended adjustment, as well as overlap and the stability of the resulting estimates. Do not select a model solely because its in-sample predictions look calibrated or because it predicts treatment labels accurately. Treatment prediction accuracy is not the same as causal adjustment quality.

The example also has important limits. Three covariate profiles and 12 units provide a transparent arithmetic exercise, not enough information to establish population performance. Grouped calibration summaries depend on the chosen groups, while model predictions can vary with the sample and tuning choices. Prediction disagreement is a prompt to examine assumptions and diagnostics, not a confidence interval for the true propensity score.