Random Forest Propensity Score Estimation

Why Use a Random Forest for a Propensity Score?

A propensity score is the probability of receiving treatment conditional on measured covariates. Write \(A_i=1\) for a treated unit and \(A_i=0\) for a control unit, and let \(X_i\) denote that unit’s covariate vector. The true propensity score is \(e(X_i)=P(A_i=1\mid X_i)\); a fitted score is an estimate of this probability. The estimation challenge is to represent how treatment assignment changes with covariates without relying on an overly restrictive functional form.

A logistic model gives predictions through a specified linear predictor, possibly expanded with transformations, interactions, or splines. A random forest instead averages predictions from many classification trees. This can represent nonlinear patterns and interactions that were not entered explicitly. The trade-off is important: the forest’s probability estimates are built from tree leaves, so they are generally piecewise constant rather than smoothly varying, and the algorithm does not guarantee good probability calibration.

A treatment model is only one part of a causal analysis. For causal interpretation using measured covariates, the usual conditions include consistency, conditional exchangeability given \(X\), and positivity: each relevant covariate pattern must have a nonzero probability of receiving either treatment. Estimating a propensity score with a forest does not establish these conditions, repair unmeasured confounding, or ensure adequate overlap.

How Bootstrap Aggregation and Randomized Features Work

A classification tree recursively divides the covariate space into regions. At a split, a candidate covariate and cut point are chosen to separate treated from control observations according to a splitting criterion, such as Gini impurity. A terminal leaf contains observations that are not split further. If \(n_{\ell}\) observations in leaf \(\ell\) are used to fit the tree and \(n_{\ell,1}\) are treated, the leaf’s estimated treatment probability is \(n_{\ell,1}/n_{\ell}\). A tree predicts that same value for every covariate vector falling in that leaf.

A forest creates variation among trees in two main ways. First, each tree is grown using a bootstrap sample: a sample of size \(n\) drawn with replacement from the \(n\) observed units. Some units can appear multiple times and some are absent from a particular tree’s sample. Second, at each candidate split, the algorithm considers a randomly selected subset of covariates rather than always searching all covariates. The number of candidate covariates is commonly called mtry. The specific split, tree structure, and leaf proportions can therefore vary across trees.

For \(B\) trees, write \(\hat e_b(X_i)\) for the treatment-probability prediction from tree \(b\). The forest prediction is

Averaging is the bootstrap-aggregation, or bagging, step. The randomized feature selection helps make trees less alike: when one strong predictor would otherwise dominate the first split in nearly every tree, restricting the candidates gives other predictors opportunities to shape some trees. Neither mechanism means that every tree is an equally good model of the treatment mechanism. They are design choices intended to produce a useful average.

Why Averaging Can Stabilize Predictions

The variance calculation below explains one benefit of averaging, while also showing why more trees cannot remove every source of variation. Suppose that, for a fixed covariate vector \(X\), the tree predictions have variance \(\sigma_T^2\), and any two distinct tree predictions have common correlation \(\rho\). These are simplifying conditions for the calculation, not claims that real trees always have identical variances and correlations. The variance of their average is

To obtain the first line, add the \(B\) individual variances and the \(B(B-1)\) ordered covariance terms, each covariance being \(\rho\sigma_T^2\), then divide by \(B^2\). If tree predictions are positively correlated, the variance approaches \(\rho\sigma_T^2\), not zero, as \(B\) grows. Bootstrap samples and randomized feature selection can reduce dependence among trees, but they do not eliminate shared information in the data or bias from a poor modeling setup.

Calibration, Smoothness, and Comparison with Logistic Predictions

A probability model should be assessed as a probability model. Calibration concerns whether units assigned probabilities near a value \(q\) experience treatment at a rate near \(q\), considered across suitable groups of units. A forest can classify treated and control observations well while still systematically overpredicting or underpredicting treatment probabilities. Classification accuracy or area under a receiver operating characteristic curve does not, by itself, establish calibration.

A logistic model with a linear predictor gives predictions of the form \(1/(1+\exp(-\eta(X)))\), where \(\eta(X)\) is the specified predictor. With finite coefficients, these predictions lie strictly between zero and one and change smoothly as \(\eta(X)\) changes. A forest prediction is an average of leaf proportions. It is generally piecewise constant over regions defined by the trees: a small covariate change may leave the prediction unchanged, while crossing a split can change it abruptly. Averaging many trees can make predictions less irregular than those of a single tree, but it does not make the forest a smooth model in the logistic-regression sense.

Neither approach is automatically better calibrated. Logistic predictions depend on whether the chosen functional form captures the treatment mechanism; a misspecified linear predictor can give systematic errors. Forest predictions can capture complex patterns but may be poorly calibrated, especially with small leaves, sparse treatment categories, or excessive adaptation to the observed sample. Calibration plots and numerical summaries should be evaluated on data not used to judge the same fitted predictions; cross-validation for model tuning provides relevant background on separating fitting from evaluation.

Practical Choices and Failure Modes

The forest’s behavior depends on tuning choices. The number of trees \(B\) controls how well the finite forest approximates its own averaging process; increasing \(B\) often reduces Monte Carlo variation in the forest prediction, but does not cure bias or guarantee calibration. The number of candidate covariates at each split affects how strongly trees resemble one another. Minimum leaf size and other restrictions affect the granularity of the fitted probabilities: very small leaves can yield noisy proportions, including values of zero or one, while larger leaves can smooth predictions by pooling more observations.

These choices should be made with the propensity-score objective in mind. A split criterion optimized for separating treated from control units need not optimize probability calibration or covariate balance for a later causal analysis. Use probability-sensitive diagnostics, inspect the distribution of estimated scores by treatment group, and check whether relevant covariate patterns have support in both groups. Calibration is not the same as balance, and neither a visually good calibration plot nor a high classification score proves that all confounders have been controlled.

The major failure modes include overly pure leaves that produce extreme estimates, sparse covariate regions where predictions depend on very few observations, and covariates that are influential but represented inconsistently across trees. A forest can also fail to represent a genuine pattern if the available data provide little information about it. As with logistic models, estimated scores near zero or one can indicate weak overlap; they deserve investigation rather than automatic acceptance or removal.

When a Forest Is a Sensible Alternative

A random forest is a useful candidate when treatment assignment may involve nonlinearities or interactions that are difficult to specify in advance, and when there are enough observations to support flexible partitions. It avoids requiring the analyst to write every interaction explicitly. Its limitations are equally practical: predictions can be locally rough, probabilities may need calibration assessment, and the fitted model is less directly summarized by a small set of coefficients than a logistic model.

A single classification tree is easier to inspect but can be unstable: modest changes in data may alter its splits and predictions. A forest reduces dependence on any one tree by averaging many varied trees. Logistic regression offers a smoother, more compact functional form when its specification is adequate, while a forest can offer greater flexibility at the cost of less transparent structure. These are alternative modeling strategies, not guarantees about which estimated score will be preferable in a particular analysis.