Why Compare Models After Weighting?
Candidate treatment models can fit observed treatment assignments differently and produce very different weights. The diagnostic question here is not which model predicts treatment most accurately. It is whether each model’s weights make the measured covariates sufficiently comparable across treatment groups while retaining enough information for a stable analysis. A model that yields attractive weights but leaves important imbalance is not automatically preferable; neither is a model with excellent balance if a few observations carry nearly all the weight.
The comparison must be fair. Use the same analysis sample, the same covariates and transformations, the same target population, and the same weight construction except for the candidate treatment model. For stabilized weights, keep the numerator model fixed across candidates. In longitudinal analyses, compare the resulting cumulative weights and check balance at the relevant decision times; a model can look acceptable at one time and perform poorly later.
Population Balance as a Reference Point
For a single binary treatment, let \(A\in\{0,1\}\), let \(H\) denote measured pretreatment covariates, and define the true treatment probability \(p(H)=P(A=1\mid H)\). A candidate model supplies \(q(H)\), with \(0<q(H)<1\) almost surely. The corresponding unstabilized inverse-probability weights are \(1/q(H)\) for treated individuals and \(1/\{1-q(H)\}\) for untreated individuals. The following identity explains why balance is a relevant comparison criterion.
Proof. Conditional on \(H\), \(E[A\mid H]=p(H)\). Moreover, \(E[|A g(H)/p(H)|\mid H]=|g(H)|\), because \(A\) is binary and \(p(H)>0\). Taking expectations gives \(E[|A g(H)/p(H)|]=E[|g(H)|]<\infty\), so the expectation exists. The law of iterated expectations now gives \(E[A g(H)/p(H)]=E[(E[A\mid H]/p(H))g(H)]=E[g(H)]\). Likewise, \(E[1-A\mid H]=1-p(H)\), and the same absolute-integrability argument gives \(E[(1-A)g(H)/\{1-p(H)\}]=E[g(H)]\). Setting \(g(H)=1\) shows that both groups’ expected weights equal one, so the corresponding normalized weighted means also equal \(E[g(H)]\). \(\square\)
For a general candidate \(q\), the same conditioning calculation gives the two weighted expectations below, provided their absolute values are integrable. A sufficient condition is that both displayed expectations of absolute values are finite; assuming merely that \(E[|g(H)|]\) is finite does not suffice when the candidate probabilities approach zero or one.
These identities concern population expectations, not guarantees about a particular finite sample. In practice, balance is estimated from the weighted data. The functions \(g\) should represent the covariate features important to the analysis: not only raw covariates, but also relevant transformations, interactions, and distributional summaries. The earlier material on standardized differences, higher moments, and empirical CDFs provides ways to examine those features.
Worked Comparisons
Worked Example: Balance Versus a Simpler Weight Distribution
Suppose there are eight observations at each of two covariate levels, \(H=0\) and \(H=1\). At \(H=0\), one is treated and three are untreated; at \(H=1\), three are treated and one is untreated. The empirical treatment probabilities are therefore \(p(0)=1/4\) and \(p(1)=3/4\). Compare a candidate model \(q_{\mathrm{strat}}(H)=p(H)\) with a constant model \(q_{\mathrm{const}}(H)=1/2\), using inverse-probability weights.
Under \(q_{\mathrm{strat}}\), the treated weighted total is \(1/(1/4)+3/(3/4)=4+4=8\). Its weighted total for \(H=1\) is \(3/(3/4)=4\), so its weighted mean of \(H\) is \(4/8=0.5\). For untreated observations, the total weight is \(3/(3/4)+1/(1/4)=4+4=8\), and the \(H=1\) weighted total is \(1/(1/4)=4\). Their weighted mean is also \(4/8=0.5\).
Under \(q_{\mathrm{const}}\), every observation has weight \(2\). The treated total is \(4(2)=8\), with \(H=1\) weighted total \(3(2)=6\), giving mean \(6/8=0.75\). The untreated weighted mean is \(1(2)/8=0.25\). Thus the constant model leaves a weighted mean difference of \(0.50\), whereas the stratified model eliminates it in this example.
The weights also reveal a trade-off. Under \(q_{\mathrm{strat}}\), treated weights are \(4\) once and \(4/3\) three times. The effective sample size, defined as \((\sum_i w_i)^2/\sum_i w_i^2\), is \(8^2/\{4^2+3(4/3)^2\}=64/(64/3)=3\). Under the constant model it is \(8^2/\{4(2^2)\}=4\). The constant model has less concentrated weights here, but its measured-covariate balance is worse. Neither diagnostic should be reported without the other.
Worked Example: Effective Sample Size and Concentration
Consider two sets of four normalized weights, each summing to 8. Candidate \(M_1\) yields \((1,1,1,5)\); candidate \(M_2\) yields \((2,2,2,2)\). For \(M_1\), the sum of squared weights is \(1+1+1+25=28\), so its effective sample size is \(8^2/28=16/7\), approximately \(2.29\). For \(M_2\), the sum of squared weights is \(4(4)=16\), so its effective sample size is \(64/16=4\).
The second candidate retains the full effective sample size in this calculation, while the first concentrates weight on one observation. This comparison alone cannot choose the model: the example gives no covariate-balance information. A model with effective sample size 4 can still leave serious imbalance, and a smaller effective sample size does not by itself prove that the weights are unusable. The weights must be assessed alongside balance and the analysis’s precision requirements.
Proof. By the Cauchy–Schwarz inequality, \((\sum_i w_i)^2\leq n\sum_i w_i^2\). Dividing by the positive quantity \(\sum_i w_i^2\) gives the bound. Equality in Cauchy–Schwarz holds exactly when the vector of weights is proportional to the vector of \(n\) ones, which means all weights are equal. \(\square\)
Worked Example: Applying Prespecified Comparison Rules
Suppose three candidate models are compared using three prespecified covariate features. For this example, each feature has a reference standard deviation of 1, so the absolute standardized mean difference equals the absolute difference between the weighted means. The diagnostic summaries are:
| Candidate | Absolute differences for the three features | Maximum difference | Treated / untreated effective sample size | Largest weight |
|---|---|---|---|---|
| \(M_1\) | 0.04, 0.07, 0.19 | 0.19 | 72 / 68 | 9 |
| \(M_2\) | 0.08, 0.06, 0.09 | 0.09 | 61 / 59 | 14 |
| \(M_3\) | 0.03, 0.04, 0.05 | 0.05 | 38 / 42 | 31 |
Assume the analysis team prespecified that all three absolute differences should be below \(0.10\), that both effective sample sizes should be at least 50, and that a largest weight above 20 warrants review. For \(M_1\), the maximum is \(0.19\), so it fails the balance rule. For \(M_2\), the maximum is \(0.09<0.10\), both effective sample sizes exceed 50, and the largest weight \(14\) is below 20; it meets all three rules. For \(M_3\), the maximum difference \(0.05\) meets the balance rule, but its effective sample sizes \(38\) and \(42\) are below 50 and its largest weight is \(31\), so it warrants concern under the prespecified criteria. On these criteria, \(M_2\) is the defensible choice among the three. This is a decision conditional on the stated features, thresholds, and analysis needs, not proof that \(M_2\) is the true treatment model.
A Practical Comparison Strategy
A disciplined comparison makes the basis for a decision explicit rather than collapsing every diagnostic into an arbitrary score. A useful sequence is:
Use the same sample, target population, weight numerator, and covariate feature set for all candidates.
Examine the prespecified covariate means and other relevant features, by treatment group and, for longitudinal weights, over follow-up and treatment histories.
Report summaries such as the effective sample size, upper tail, and maximum weight, and check whether a small number of observations dominate.
Identify which candidates meet the balance and weight requirements; explain any choice without treating a single summary as decisive.
Likelihood and prediction measures answer different questions. Likelihood assesses how well a model represents observed treatment assignments under its probability model; prediction discrimination assesses how well it separates treated from untreated observations. Neither directly establishes that weighting has balanced the covariates relevant to the causal analysis. A candidate can predict treatment well while leaving weighted imbalance, and a less discriminating model can sometimes produce more useful balance. This is why post-weighting diagnostics, rather than fit or discrimination alone, are central to this comparison.
Diagnostics, Pitfalls, and Limits
Look for residual imbalance in individual features, transformations, or distributional comparisons, not only in an overall average. In longitudinal settings, inspect whether imbalance reappears at later times or within important treatment histories. Also check whether the apparent balance is being achieved with very few effective observations or a highly concentrated upper tail. A diagnostic that looks satisfactory only after changing the analysis sample or the balance features for one candidate is not a fair comparison.
- Pitfall: choosing the highest likelihood or best discrimination. That ranks prediction performance, not post-weighting comparability. Compare weighted balance directly.
- Pitfall: choosing the smallest maximum imbalance automatically. A model can minimize that number while producing highly concentrated weights. Examine both balance and weight behavior.
- Pitfall: declaring a model correct because measured covariates balance. Balance on observed features cannot verify that all confounding has been measured or that the causal assumptions hold.
- Pitfall: comparing incompatible weights. Changing the target population or stabilization numerator across candidates can make differences hard to interpret. Keep these fixed when the aim is to compare treatment models.
No universal balance threshold or effective-sample-size cutoff is suitable for every study. Thresholds should be set with the covariate scales, sample size, estimand, and intended outcome analysis in mind. These diagnostics are also finite-sample summaries: they do not establish the population identity, prove positivity, or guarantee unbiased effect estimates. They help identify and compare measured-data consequences of candidate models; they cannot certify causal validity.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Why can a candidate treatment model with the best treatment prediction still be a poor choice for weighting?
- For weights \((1,1,2)\), calculate the effective sample size.
- Under the correct treatment model, why does the treated weighted expectation of \(g(H)\) equal \(E[g(H)]\)? State the needed integrability condition.
- One candidate has better balance but much lower effective sample size than another. What should be examined before choosing?
Solutions. (1) Prediction fit does not directly show whether the weights balance measured covariates; compare post-weighting balance and weight concentration. (2) The weights sum to \(4\), and their squares sum to \(1+1+4=6\), so the effective sample size is \(4^2/6=8/3\). (3) If \(q=p\) and \(E[|g(H)|]<\infty\), then conditional expectation gives \(E[A g(H)/p(H)]=E[(E[A\mid H]/p(H))g(H)]=E[g(H)]\); the absolute expectation is finite because \(E[|A g(H)/p(H)|]=E[|g(H)|]\). (4) Check whether the better-balanced candidate meets the analysis’s precision needs, inspect its upper-tail weights and influential observations, and apply comparison criteria chosen for the target population and estimand. Balance alone does not settle the decision.