Why Ensemble Propensity Estimators?
A propensity model estimates the probability of treatment assignment given measured pretreatment covariates. Different model families can represent different patterns: logistic regression offers a structured parametric form, smooth models can represent nonlinear covariate relationships, and tree-based estimators can capture interactions and threshold-like patterns. Before analysis, it is often unclear which set of modeling choices will best predict treatment assignment in new observations.
Ensembling addresses this uncertainty by combining predictions from several candidate estimators. Rather than choosing one model in advance, a stacking procedure uses cross-validated predictions to learn how to weight the candidates. The goal here is a probability estimate for treatment assignment. A strong predictive ensemble can be useful, but predictive accuracy by itself does not demonstrate that a causal analysis is valid or that covariates are balanced after matching or weighting.
For a causal interpretation, the covariates used in the analysis must be appropriate pretreatment variables, and the causal assumptions underlying propensity-score methods must be considered. In particular, a fitted treatment probability does not itself establish consistency, conditional exchangeability, or positivity. Those assumptions concern the study and its treatment assignment process; changing the prediction algorithm cannot make them true.
From Candidate Predictions to an Ensemble Probability
Suppose there are \(M\) candidate propensity estimators. For a given unit \(i\), let \(\hat e_{im}\) be the probability predicted by candidate \(m\). A constrained linear stack assigns each candidate a weight \(w_m\) and combines its predicted probabilities:
Because every candidate probability lies in \([0,1]\), and the weights are nonnegative and sum to one, the ensemble prediction also lies in \([0,1]\). It is therefore a probability without needing a further transformation. The weight constraint also prevents a candidate from receiving a negative contribution or an arbitrarily large coefficient that pushes the result outside the probability range.
Worked Example: Combining Three Candidate Predictions
For one hypothetical patient, suppose logistic regression predicts a treatment probability of 0.30, a smooth model predicts 0.50, and a tree-based estimator predicts 0.80. Suppose cross-validated stacking has selected weights 0.50, 0.30, and 0.20, respectively. The weights are nonnegative and sum to one.
The ensemble probability is
The resulting estimate is a 0.46 probability of treatment for a unit with these covariates, according to this fitted ensemble. It is not a statement that this particular patient has a 46% chance of benefiting from treatment; the propensity score concerns treatment assignment, not treatment response.
Choosing Weights With Cross-Validated Predictions
Weights are commonly chosen using log loss, a measure of how well predicted probabilities correspond to observed treatment assignments. For a prediction \(q_i\) and observed treatment \(A_i\), the binary log loss is
A confident prediction that assigns a very small probability to the treatment actually observed receives a large penalty. For example, when \(A_i=1\), the loss is \(-\log(q_i)\); when \(A_i=0\), it is \(-\log(1-q_i)\). Lower average log loss indicates better probabilistic predictions on the evaluated observations. In software, predictions are often bounded away from exactly 0 and 1 before taking logarithms, to avoid infinite numerical values.
To estimate stacking weights, divide the observations into \(K\) folds. For each fold, fit every candidate estimator using the other folds, then predict treatment probabilities for observations in the held-out fold. This gives each observation a prediction from each candidate that was not fitted using that observation. Select the weights by minimizing the average log loss of the weighted predictions, subject to the nonnegative weights summing to one.
Here \(\hat e_{im}^{\mathrm{CV}}\) is candidate \(m\)'s cross-validated prediction for unit \(i\). The weights are common across observations in this basic stack; the candidate predictions vary from unit to unit. Once weights are estimated, a standard implementation fits each candidate to the full available sample and combines its predictions for the units of interest using those weights. This final refitting step does not change the definition of the cross-validated predictions used to learn the weights.
Why Log Loss Is a Coherent Choice
Log loss is a proper scoring rule for probabilities: in expectation, the true conditional treatment probability minimizes the expected loss. To see this, condition on covariates \(X_i=x\), and write \(e=P(A_i=1\mid X_i=x)\). If a proposed probability is \(q\), the conditional expected loss is
For \(0<q<1\), differentiate with respect to \(q\):
The denominator is positive, so the derivative is negative when \(q<e\) and positive when \(q>e\). Thus the loss decreases up to \(q=e\) and increases after it; the minimum is at the true treatment probability. This explains why log loss is a sensible criterion for probabilistic prediction. It does not show that the lowest-loss estimated model has recovered the true probabilities exactly in a finite sample.
For the constrained weighted stack, each predicted probability is an affine function of the weights. The negative logarithm of a positive affine function is convex, and the same is true of the untreated term, which uses one minus an affine probability. Therefore the average log loss is convex over weights for which predictions are strictly between 0 and 1. The feasible weights form a convex simplex. This structure supports stable numerical optimization, although finite samples can still yield uncertain or boundary weights.
Worked Example: Comparing Held-Out Log Losses
Consider two hypothetical held-out observations, both of whom received treatment. A reference candidate predicts treatment probabilities 0.55 and 0.65. A fitted ensemble predicts 0.55 and 0.6546. Since both observed values of \(A_i\) equal 1, each loss is the negative logarithm of its predicted treatment probability.
For the reference candidate, the mean loss is
For the ensemble, the mean loss is
The ensemble's held-out loss is lower by approximately \(0.5143-0.5108=0.0035\) log-loss units per observation in this tiny example. This is a numerical comparison, not evidence that the ensemble will improve prediction in a new sample: two observations provide little information, and a separate assessment is needed to judge generalization.
What the Weights Do—and Do Not—Guarantee
The optimization selects weights that perform well according to the chosen loss on cross-validated predictions. It does not guarantee that every candidate contributes a positive weight, that the ensemble will beat the best candidate on new data, or that the resulting estimated scores produce exact sample balance. A weight of zero simply means that the optimizer did not use that candidate in the fitted combination under the chosen criterion and constraints.
A useful diagnostic is to inspect the weights alongside predictive performance, the distribution of estimated probabilities, and covariate balance relevant to the planned analysis. If one candidate receives nearly all the weight, the procedure may effectively have selected that candidate. If several candidates receive weight, the combination can still be dominated by similar predictions. A lower average loss may coexist with extreme estimated probabilities or poor balance on important covariates.
Cross-validation reduces the direct optimism that would arise from fitting candidates and evaluating them on the same observations. However, if the same cross-validated predictions are used both to choose weights and to report the ensemble's performance, the reported performance can still be optimistic because the weights were tuned to those predictions. A separate assessment or an appropriately nested validation design is useful when an unbiased estimate of predictive performance is important. The choice of folds, random partition, candidate specifications, and loss function can also affect the result.
Common Mistakes and Alternatives
| Approach or mistake | What it does | What to keep in mind |
|---|---|---|
| Hard voting | Turns each candidate prediction into a treatment/no-treatment class, then combines class votes. | It discards probability information, so it is generally not suitable when the desired output is a propensity probability. |
| Average logits | Averages candidate log-odds and transforms the result back to a probability. | This differs from averaging probabilities; it can emphasize predictions near 0 or 1 differently. |
| Choose one candidate by validation loss | Selects the single candidate with the lowest validation loss. | It is simpler, but may be sensitive to validation variation and does not combine complementary predictions. |
| Assume a low loss proves causal validity | Treats accurate treatment prediction as evidence that causal assumptions hold. | Prediction quality cannot establish absence of unmeasured confounding or other causal assumptions. |
Probability averaging is a direct way to keep the ensemble output on the probability scale. Other stacking formulations, such as a logistic meta-model, can also be used, but they define a different combination and require their own validation. Regardless of the chosen form, the outcome being predicted in this task is \(A_i\), not a potential outcome. The ensemble is a treatment-assignment model, not an outcome model.
Worked Example: A Mixed-Treatment Validation Pair
For two hypothetical held-out units, the first treated and the second untreated, suppose candidate 1 predicts probabilities \((0.80,0.70)\) and candidate 2 predicts \((0.60,0.20)\). For illustration, assign each candidate weight 0.50. The ensemble probabilities are
The first unit was treated, so its loss is \(-\log(0.70)=0.356675\). The second was untreated, so its loss is \(-\log(1-0.45)=-\log(0.55)=0.597837\). The average ensemble loss is
For comparison, candidate 1's mean loss is \(\{ -\log(0.80)-\log(1-0.70)\}/2=(0.223144+1.203973)/2\approx0.7136\). Candidate 2's mean loss is \(\{-\log(0.60)-\log(1-0.20)\}/2=(0.510826+0.223144)/2\approx0.3670\). Thus this equal-weight ensemble improves on candidate 1 but not candidate 2 for these two observations. Ensembling is not guaranteed to outperform every candidate, and weights should be learned and assessed using a suitable validation design rather than assumed to be equal.
Check Your Understanding
Try each question, then compare your reasoning with the solutions below.
- A logistic model predicts 0.25 and a smooth model predicts 0.65 for the same unit. With weights 0.75 and 0.25, what is the ensemble probability?
- For a treated unit with predicted treatment probability 0.80, write its log loss and give its approximate value.
- Why do stacking procedures use predictions for observations not used to fit the corresponding candidate model?
- Does a lower cross-validated treatment-prediction loss establish that conditional exchangeability holds? Explain.
- What is one difference between probability averaging and hard voting?
Solutions. (1) The weights sum to one, and \(0.75(0.25)+0.25(0.65)=0.1875+0.1625=0.35\). (2) Because the unit is treated, the loss is \(-\log(0.80)\approx0.2231\) log-loss units. (3) Held-out predictions limit the direct reuse of each observation for both fitting a candidate and evaluating its prediction, which would otherwise make performance appear overly favorable. (4) No. Lower loss describes predictive performance for treatment assignment; conditional exchangeability is a causal assumption and cannot be verified from that comparison alone. (5) Probability averaging retains each candidate's numerical probability, whereas hard voting first converts probabilities into classes and discards information about their magnitudes.