Why Compare Treatment-Model Ensembles?
A treatment model is a working model for the probability of receiving the observed treatment given measured covariate history. Different learners can represent that relationship in different ways: a logistic regression with specified terms, a model with prespecified interactions or nonlinear terms, or a flexible algorithm. An ensemble combines predictions from multiple learners. The purpose here is not to declare an ensemble automatically superior; it is to ask whether the weights and causal estimates are sensitive to reasonable, prespecified modeling choices.
The earlier material on comparing effect estimates across candidate models motivates looking beyond one fitted model. An ensemble adds another comparison: does a prespecified combination of learners yield meaningfully different predictions, weights, balance, or effect estimates? Stability across these analyses is useful diagnostic evidence, but it cannot establish that the treatment mechanism is correctly modeled or that causal assumptions hold.
For a simplified longitudinal setting with binary treatment, define a stabilized treatment weight by
Here \(\hat p_{it}\) is the fitted denominator probability for the treatment actually received, and \(\hat q_{it}\) is the fitted numerator probability under the chosen stabilization model. For a single treatment decision with numerator \(P(A=1)=0.5\), the treated weight is \(0.5/\hat e_i\), and the untreated weight is \(0.5/(1-\hat e_i)\). These formulas define the weights; a causal interpretation of weighted estimates additionally requires consistency, sequential exchangeability given measured history, and positivity for the treatment strategies being compared. Comparing learners does not verify these assumptions.
What an Ensemble Does—and Does Not Do
A convex combination is convenient because it remains on the probability scale. But averaging predictions is not the same operation as averaging weights or effect estimates: denominators are inverted to form weights, and weighted means are ratios. Consequently, the ensemble’s effect estimate need not equal the average of the candidate estimates.
To prove the result, write \(\epsilon\leq\hat e^{(k)}\leq1-\epsilon\) for every \(k\). Multiplying each inequality by \(\alpha_k\geq0\) and summing gives \(\epsilon\sum_k\alpha_k\leq\sum_k\alpha_k\hat e^{(k)}\leq(1-\epsilon)\sum_k\alpha_k\). Since the coefficients sum to one, \(\epsilon\leq\hat e^{(E)}\leq1-\epsilon\). The same argument with bounds zero and one proves the first claim. Therefore both \(1/\hat e^{(E)}\) and \(1/(1-\hat e^{(E)})\) are at most \(1/\epsilon\). This is a numerical boundedness result, not a causal-validity result: even bounded probabilities can be misspecified, and a positivity assumption about the data-generating process is not established by clipping or averaging predictions. \(\square\)
Worked Example: Candidate Learners Give Different Estimates
There are four individuals, with treatment \(A=(1,1,0,0)\) and outcome \(Y=(2,8,0,10)\). Use numerator probability \(0.5\). Learner 1 predicts treatment probabilities \((0.8,0.6,0.2,0.2)\); learner 2 predicts \((0.6,0.6,0.4,0.4)\). For each learner, calculate the inverse-probability weighted mean in each treatment group and their difference.
For learner 1, the treated weights are \(0.5/0.8=0.625\) and \(0.5/0.6=0.8333\). The treated weighted mean is \((0.625(2)+0.8333(8))/(0.625+0.8333)=7.9167/1.4583\approx5.429\). The two untreated weights are both \(0.5/0.8=0.625\), because the predicted treatment probability is \(0.2\). Their weighted mean is \((0.625(0)+0.625(10))/1.25=5\). The estimated difference is approximately \(5.429-5=0.429\).
For learner 2, each observed-treatment probability is \(0.6\): this is \(0.6\) for treated individuals and \(1-0.4=0.6\) for untreated individuals. All four weights are \(0.5/0.6=0.8333\). The treated mean is \((2+8)/2=5\), the untreated mean is \((0+10)/2=5\), and the estimated difference is zero. Thus, two reasonable-looking probability vectors produce different weighted estimates. This comparison signals sensitivity; it does not tell us which learner is correct.
Worked Example: An Ensemble Estimate Is Not an Average of Estimates
Use the data and predictions from the preceding calculation, and form the equal-weight ensemble, with \(\alpha_1=\alpha_2=0.5\). The ensemble predictions are \((0.7,0.6,0.3,0.3)\). Calculate the ensemble estimate and compare it with the average of the two candidate estimates.
The treated weights are \(0.5/0.7=5/7\) and \(0.5/0.6=5/6\). Therefore the treated weighted mean is \((2(5/7)+8(5/6))/(5/7+5/6)=(170/21)/(65/42)=68/13\approx5.231\). For each untreated individual, the weight is \(0.5/(1-0.3)=5/7\). The untreated weighted mean is \((0(5/7)+10(5/7))/(10/7)=5\). The ensemble difference is \(68/13-5=3/13\approx0.231\).
The candidate differences were approximately \(0.429\) and \(0\); their average is approximately \(0.214\), not \(0.231\). The discrepancy occurs because the weighted means use ratios whose denominators change with the predictions. The ensemble’s estimate must therefore be calculated from its own weights rather than inferred by averaging candidate estimates.
What Estimate Stability Can Tell You
The following identity makes precise one way weight changes affect a weighted mean. Consider fixed outcomes \(Y_i\), nonnegative weights \(w_i\) and \(w_i'\), and positive sums \(S=\sum_i w_i\) and \(S'=\sum_i w_i'\). Define \(\hat\mu(w)=\sum_iw_iY_i/S\) and \(\hat\mu(w')=\sum_iw_i'Y_i/S'\). Then
To derive the identity, first use \(\sum_iw_i\{Y_i-\hat\mu(w)\}=0\), which follows directly from the definition of \(\hat\mu(w)\). Next, \(\sum_iw_i'\{Y_i-\hat\mu(w)\}=\sum_i(w_i'-w_i)\{Y_i-\hat\mu(w)\}+\sum_iw_i\{Y_i-\hat\mu(w)\}\). The second sum is zero. Dividing the left side by \(S'\) gives \(\hat\mu(w')-\hat\mu(w)\), proving the identity. In particular,
This follows from the triangle inequality and \(S'>0\). The bound shows why a small number of observations can drive sensitivity when their weights change substantially and their outcomes differ from the original weighted mean. For a treatment effect, this reasoning applies to each treatment-specific weighted mean; changes in both groups contribute to changes in the contrast.
Worked Example: Product Weights Can Amplify Model Differences
For one individual observed at two treatment times, suppose the numerator probabilities for the observed treatment are \(0.5\) at each time. Learner M assigns denominator probabilities \(0.8\) at time 1 and \(0.3\) at time 2 for the treatments actually received. Learner N assigns \(0.6\) and \(0.5\). Calculate both stabilized weights.
For learner M, the two time-specific factors are \(0.5/0.8=0.625\) and \(0.5/0.3=5/3\). Their product is \(0.625(5/3)=25/24\approx1.042\).
For learner N, the factors are \(0.5/0.6=5/6\) and \(0.5/0.5=1\), so the product is \(5/6\approx0.833\). The predictions differ at both times, and multiplying the factors changes the resulting weight. This calculation is for one individual only; a practical comparison should examine the full weight distributions, not infer overall stability from a single subject.
A Practical Sensitivity Comparison
For a useful comparison, define the candidate learners and ensemble procedure in advance, then fit each using the same analysis sample and the same numerator specification. If the fitting procedure uses data-adaptive selection, predictions used for diagnostics should be produced in a way that addresses overfitting, such as out-of-fold prediction. This does not make the resulting model correct; it helps prevent an overly optimistic assessment from predictions evaluated on the same observations used to fit them.
Record the learner library, prediction scale, ensemble coefficients or selection rule, and any probability-truncation rule. Do not choose a combination solely because it produces a preferred effect estimate.
Inspect predicted treatment probabilities, stabilized-weight tails, the largest weights, and effective sample size. Repeat the comparison across treatment times when weights are products.
Assess post-weighting balance using the same covariate summaries and compare the target effect estimates and their uncertainty. A stable estimate accompanied by poor balance is not reassuring.
Trace large changes to probability predictions, influential observations, or weight tails. Report sensitivity transparently rather than selecting the learner that gives the most attractive result.
Weight distributions, effective sample size, and post-weighting balance are complementary checks. No single diagnostic certifies the treatment model. In particular, good discrimination does not guarantee calibrated probabilities or balance, and an ensemble cannot repair a lack of treatment support in the data. If extreme weights appear, truncating them may change the estimate and target of the weighted procedure; it should be reported as an additional analysis, not presented as proof that the original problem has been solved. Likewise, agreement among learners that share the same omitted confounder or structural error is not strong evidence against that shared failure.
The principal alternative is to select one prespecified working model and report its diagnostics, possibly alongside a limited set of sensitivity specifications. An ensemble can reduce reliance on a single learner’s functional form, but introduces choices about its library, fitting, and combination rule. Neither approach dominates automatically. The relevant evidence is how well the resulting probabilities support the intended weighting analysis, how balance and weights behave, and how conclusions change across defensible choices. These checks address model sensitivity; they do not establish exchangeability or identify a causal effect without the required assumptions.
Common Mistake
Wrong reasoning: “The ensemble averages several treatment models, so it must be less misspecified and its estimate is a reliable causal answer.” Averaging may smooth differences, but it cannot guarantee that the candidates include an adequate model, that measured covariates suffice for exchangeability, or that positivity holds. Further, the ensemble estimate is not generally the average of candidate estimates.
Correction: Treat the ensemble as one prespecified sensitivity analysis. Calculate its own predictions, weights, balance, and estimate; compare these with the candidates; and state what the diagnostics do and do not show. Large instability calls for investigation, while stability is supportive but not conclusive evidence.
Check Your Understanding
Answer each question, then check the solutions below.
- Why does a convex ensemble of predictions bounded between \(\epsilon\) and \(1-\epsilon\) also remain within those bounds?
- In the four-person example, what were the ensemble’s treated mean, untreated mean, and estimated difference?
- Why is the ensemble effect estimate not generally the average of the candidate effect estimates?
- What does agreement across learners establish about exchangeability and positivity?
- If two models give similar effect estimates but one yields poor post-weighting balance, what is the appropriate interpretation?
Solutions. (1) Multiply each candidate’s lower and upper bounds by its nonnegative ensemble coefficient and sum; the coefficients sum to one, so the bounds remain \(\epsilon\) and \(1-\epsilon\). (2) The treated mean is \(68/13\approx5.231\), the untreated mean is \(5\), and the difference is \(3/13\approx0.231\). (3) Weighted estimates are ratios, and averaging probabilities, inverting them to obtain weights, and normalizing weighted outcomes do not generally commute. (4) Nothing definitive: agreement is a sensitivity observation and does not prove either causal assumption. (5) The estimate agreement is not sufficient reassurance; report the poor balance and investigate the model and weights before interpreting the effect causally.