Comparing Effect Estimates Across Candidate Models

Why Compare Effect Estimates Across Candidate Models?

Alternative treatment models can produce different inverse-probability weights and, consequently, different causal effect estimates. A comparison is informative only if it helps explain that variation. In particular, an estimate may change because the candidate models produce different covariate balance, different weight concentration, or different weighted outcome distributions. These features are related, but none is a substitute for the others.

The comparison should hold the target estimand and the rest of the analysis fixed: use the same observations, outcome definition, treatment contrast, weight convention, truncation rule (if any), and effect estimator. Otherwise, the difference between estimates mixes treatment-model sensitivity with changes to the analysis itself. In longitudinal settings, the same principle applies to the treatment history, follow-up, and marginal structural model.

Definition: A candidate-model comparison computes the same causal effect estimator under two or more specified treatment models, then examines the resulting estimates together with model-specific weight diagnostics. For a binary treatment and an average treatment effect estimated by inverse-probability weighting, a common estimator is the difference between the weighted outcome mean among treated individuals and that among untreated individuals.

For candidate model $m$, let $w_{im}$ denote individual $i$'s weight, $A_i$ the observed binary treatment, and $Y_i$ the outcome. With positive weights, the weighted mean in treatment group $a$ is

\hat\mu_{a,m} = \frac{\sum_{i=1}^{n} I(A_i=a)w_{im}Y_i} {\sum_{i=1}^{n} I(A_i=a)w_{im}}, \qquad \hat\tau_m=\hat\mu_{1,m}-\hat\mu_{0,m}.

Here $I(\cdot)$ is an indicator. For the usual unstabilized inverse-probability weights, $w_{im}=1/\hat e_m(X_i)$ when $A_i=1$, and $w_{im}=1/\{1-\hat e_m(X_i)\}$ when $A_i=0$, where $\hat e_m(X_i)$ is the treatment probability estimated by candidate model $m$. Stabilization can multiply weights by group-specific constants; those constants cancel from these group-specific weighted means.

What Must Be Held Fixed, and What Can Be Learned?

A causal interpretation requires the usual identification conditions for the estimand: consistency, exchangeability given the measured covariates used in the treatment model, and positivity for the treatment strategies being contrasted. The weights must also be constructed for the intended estimand. For example, weights for an average treatment effect differ from weights targeting a treated population. Under these conditions and a correctly specified treatment model, the weighted estimator can identify the same target under each candidate specification. If models are misspecified, their weighted estimators may instead converge to different biased or otherwise model-dependent quantities.

Candidate comparisons are therefore a sensitivity analysis, not a proof that any model is correct. The earlier material on post-weighting balance and treatment-model sensitivity motivates examining balance alongside estimates. A small change in an estimate is not evidence of adequate balance; a large change does not identify which model is better. Weight concentration matters too, because a weighted estimate can be dominated by a few observations even when a reported balance summary looks acceptable.

Effective sample size: For nonnegative weights $w_1,\ldots,w_k$ in one analysis group, the effective sample size is $$ \operatorname{ESS}=\frac{\left(\sum_{i=1}^{k}w_i\right)^2}{\sum_{i=1}^{k}w_i^2}. $$ It summarizes weight concentration on a scale from 1 to $k$, but it is not a literal count of independent observations and does not measure balance.

Worked Example: Two Models, Similar Balance Magnitude, Different Estimates

Consider six individuals: one treated and one untreated individual at each value $X=0,1,2$. The treated outcomes at those values are $2,4,8$, and the untreated outcomes are $0,2,4$. Candidate model A assigns treatment probabilities $1/4,1/2,3/4$ at $X=0,1,2$; candidate model B assigns $3/4,1/2,1/4$. These are positive probabilities. The resulting unstabilized inverse-probability weights, listed in order $X=0,1,2$, are:

\begin{array}{c|cc} & A_i=1 & A_i=0\\ \text{Model A}&(4,2,4/3)&(4/3,2,4)\\ \text{Model B}&(4/3,2,4)&(4,2,4/3) \end{array}

Each arm has total weight $4+2+4/3=22/3$ under either model. For model A, the treated weighted mean is $(4(2)+2(4)+(4/3)(8))/(22/3)=(8+8+32/3)/(22/3)=40/11$. The untreated weighted mean is $((4/3)(0)+2(2)+4(4))/(22/3)=20/(22/3)=30/11$. Thus $\hat\tau_A=10/11$.

For model B, the treated weighted mean is $((4/3)(2)+2(4)+4(8))/(22/3)=(8/3+8+32)/(22/3)=64/11$. The untreated weighted mean is $(4(0)+2(2)+(4/3)(4))/(22/3)=(4+16/3)/(22/3)=14/11$. Thus $\hat\tau_B=50/11$. The difference between estimates is $40/11$, despite the models having the same total weight in each arm.

For either arm under either model, the squared weights sum to $4^2+2^2+(4/3)^2=16+4+16/9=196/9$. Therefore $\operatorname{ESS}=(22/3)^2/(196/9)=484/196=121/49$, about $2.47$, in each arm. The weighted mean of $X$ among treated individuals is $7/11$ under A and $15/11$ under B; among untreated individuals it is $15/11$ under A and $7/11$ under B. The absolute difference in these weighted means is $8/11$ under both models. Thus equal ESS and equal absolute imbalance in this covariate do not imply equal effect estimates.

Decomposing the Difference Between Estimates

A useful first step is to compare the weighted outcome mean separately in each treatment group. This identifies which arm contributes to the overall change; it does not by itself explain why that arm's weighted mean changed. The following identity makes that accounting exact.

Result: For any two candidate models A and B using the same weighted-mean estimator, $$ \hat\tau_B-\hat\tau_A = (\hat\mu_{1,B}-\hat\mu_{1,A}) - (\hat\mu_{0,B}-\hat\mu_{0,A}). $$

Proof. By definition, $\hat\tau_B=\hat\mu_{1,B}-\hat\mu_{0,B}$ and $\hat\tau_A=\hat\mu_{1,A}-\hat\mu_{0,A}$. Subtracting gives \[ \hat\tau_B-\hat\tau_A =(\hat\mu_{1,B}-\hat\mu_{0,B})-(\hat\mu_{1,A}-\hat\mu_{0,A}) =(\hat\mu_{1,B}-\hat\mu_{1,A})-(\hat\mu_{0,B}-\hat\mu_{0,A}). \] This is an algebraic identity; it does not require either model to be correct. $\square$

Worked Example: Locating the Source of an Estimate Change

Use the estimates from the preceding example. The treated weighted mean changes from $40/11$ under A to $64/11$ under B, a change of $24/11$. The untreated weighted mean changes from $30/11$ to $14/11$, a change of $-16/11$. Therefore

\hat\tau_B-\hat\tau_A = \frac{24}{11}-\left(-\frac{16}{11}\right) = \frac{40}{11}.

Direct subtraction gives $50/11-10/11=40/11$, which checks the decomposition. Both arms contribute: the treated mean rises, while the untreated mean falls. This accounting isolates where the numerical change enters the contrast, but it does not establish that either weighted mean is causally unbiased.

Result: For $k$ nonnegative weights with positive sum, $1\leq\operatorname{ESS}\leq k$. Multiplying all weights by the same positive constant leaves ESS unchanged.

Proof. Write $S=\sum_{i=1}^{k}w_i>0$ and $Q=\sum_{i=1}^{k}w_i^2$. Since $w_i\geq0$, $S^2=Q+2\sum_{i<j}w_iw_j\geq Q$, so $S^2/Q\geq1$. By the Cauchy–Schwarz inequality, $S^2\leq kQ$, so $S^2/Q\leq k$. If every weight is replaced by $cw_i$, with $c>0$, the new ESS is $(cS)^2/(c^2Q)=S^2/Q$. $\square$

The scale invariance explains why ESS can remain constant when a model changes only the overall scale of weights within an arm. It also cautions against treating ESS as a diagnostic of balance: its formula contains no covariate values or outcomes.

Worked Example: Better Balance and Higher ESS Do Not Determine the Estimate

In the same six-person data, consider candidate model C with estimated treatment probability $1/2$ at every value of $X$. Every unstabilized weight is then $2$. Each arm's weighted mean of $X$ is the ordinary mean $(0+1+2)/3=1$, so the difference in weighted means is zero. Each arm has ESS $(2+2+2)^2/(2^2+2^2+2^2)=36/12=3$.

The treated weighted outcome mean is $(2(2)+2(4)+2(8))/6=28/6=14/3$. The untreated mean is $(2(0)+2(2)+2(4))/6=12/6=2$. Thus $\hat\tau_C=14/3-2=8/3$. This estimate differs from both $\hat\tau_A=10/11$ and $\hat\tau_B=50/11$; its exact balance on the displayed $X$ and its larger ESS do not establish that C is correctly specified. Nor do they establish that the other models are correct. The example illustrates why estimate, balance, and ESS should be reported and interpreted as distinct pieces of evidence.

Practical Comparison and Failure Modes

A disciplined comparison begins by tabulating, for each candidate, the effect estimate, treatment-arm-specific weighted outcome means, relevant post-weighting balance summaries, and ESS in each treatment group. Use the same covariate summaries and standardization convention across models. In longitudinal analyses, preserve the same time-specific or history-specific balance checks and the same definition of the final estimand. Candidate-specific changes in the weights may be examined, but they should not be confused with changes caused by changing truncation or the analysis sample.

If the point estimate changes substantially, inspect the treatment-arm means first, then examine whether the change accompanies residual imbalance, concentrated weights, or both. Check weight distributions and influential observations as well as summary ESS. The earlier material on influence analysis is relevant: an arm-level ESS can look reasonable while a particular observation still has strong leverage. Conversely, a lower ESS does not prove that the estimate is biased; it signals that the weighted mean is supported by less broadly distributed weight.

A competing approach is to select a single candidate using treatment-model fit or discrimination. Those summaries alone do not show whether the weighted pseudo-population has the desired balance, and choosing the model with the most favorable effect estimate is especially problematic. Comparing model-specific weighted estimates is more informative as a sensitivity analysis, but it does not replace substantive specification or identification assumptions. Outcome-regression approaches can provide a different analysis, but a comparison with them is meaningful only after clarifying whether they target the same estimand and what assumptions they require.

Changing several ingredients at once: If one candidate also changes the target population, truncation threshold, or sample, the estimate difference cannot be attributed solely to treatment-model specification.
Equating balance with agreement: Similar balance on selected summaries can coexist with different weighted outcome means, as the first example shows. Diagnostics cover observed covariates and summaries, not every possible source of misspecification.
Equating ESS with precision or validity: ESS is a weight-concentration summary. It does not account by itself for outcome variability, dependence, estimated-weight uncertainty, or unmeasured confounding.
Choosing the preferred estimate: Selecting a model because its estimate is closer to a desired result is not a diagnostic rule. Report the sensitivity and justify model choices using design knowledge and diagnostics, not the preferred causal conclusion.

Finally, a comparison of point estimates is not a comparison of their sampling uncertainty. Inference must account for the fact that treatment probabilities and weights are estimated, and for any clustering or repeated observations in the study design. If uncertainty across candidate-model estimates is of interest, the entire estimation procedure must be represented in the uncertainty analysis; treating estimated weights as fixed can understate uncertainty.

Fix the target and estimator.
Specify the same population, treatment contrast, outcome, weight convention, and analysis sample for every candidate.

Compute the full comparison.
Record each effect estimate and its two weighted arm means, with balance summaries and arm-specific ESS.

Diagnose the differences.
Check whether estimate changes accompany altered balance or weight concentration; inspect weight distributions and influential observations.

State the limitation.
Interpret disagreement as sensitivity to specification, not as proof that one candidate is correct or that a particular estimate is the truth.

Check Your Understanding

Answer each question, then check the solutions below.

Why must the target estimand and truncation rule be held fixed when attributing estimate changes to candidate treatment models?
If two candidates have the same ESS in each arm, what does that establish, and what does it not establish?
In a comparison, the treated mean rises by $0.6$ and the untreated mean rises by $0.2$. By how much does the estimated treatment contrast change?
Can a model with better measured-covariate balance be declared correct from that fact alone? Explain.
What does ESS summarize, and name one important feature it does not measure.

Solutions. (1) Otherwise the estimates differ partly because the target population or weight processing changed, so the comparison does not isolate model specification. (2) It establishes equal values of the particular weight-concentration summary, not equal covariate balance, weighted outcome means, estimator variance, or causal validity. (3) The change is $0.6-0.2=0.4$, because the contrast is treated mean minus untreated mean. (4) No. Balance is diagnostic evidence about measured covariates; it does not establish correct model specification, exchangeability, positivity, or absence of unmeasured confounding. (5) ESS summarizes concentration of nonnegative weights. It does not measure balance, outcome variability, or causal validity.

Comparing Effect Estimates Across Candidate Models

What You'll Learn