Why ranking is not weighting
A treatment model can separate treated from untreated people very well and still assign probabilities that produce poor inverse probability weights. The reason is that discrimination asks a ranking question—does a treated person tend to receive a higher score than an untreated person?—whereas weighting depends on the numerical probability values. A model may preserve the ranking while substantially misrepresenting those values.
This distinction matters because inverse probability weights use reciprocals of treatment probabilities. A small error in a probability near zero or one can therefore have a large effect on a weight. In contrast, the area under the receiver operating characteristic curve (AUC) is unchanged by many alterations to the score that can change those reciprocals greatly.
The examples below use a single binary treatment \(A\) and baseline covariates \(L\). Write the true propensity score as \(e(L)=P(A=1\mid L)\), and let \(q(L)\) denote the probability assigned by a fitted treatment model. For causal interpretation of a weighted outcome analysis, additional conditions are needed: consistency, conditional exchangeability \(Y^a \perp A\mid L\), positivity, and an adequately specified treatment probability model. The weight identities below are distributional facts; by themselves, they do not establish those causal conditions.
A ranking property of AUC
The following property explains why discrimination can remain unchanged even when the numerical propensities—and hence the weights—change. The transformation must be strictly increasing so that it preserves both ordering and ties.
Proof. For any two possible covariate values, strict increase gives \(s(L_1)>s(L_0)\) if and only if \(g(s(L_1))>g(s(L_0))\). It also gives \(s(L_1)=s(L_0)\) if and only if \(g(s(L_1))=g(s(L_0))\). Thus the event of a strict score win and the event of a tie are unchanged. Their probabilities, and therefore the AUC, are unchanged. \(\square\)
For example, a model that reports probabilities \(0.20\) and \(0.80\) for two covariate groups ranks them in the same order as a model that reports \(0.02\) and \(0.98\). The AUC cannot distinguish these two sets of values when the ordering and ties are the same. But the inverse weights can be very different. This is one reason a high AUC does not certify a useful weighting model.
What correct probabilities do for weights
For a point treatment, the unstabilized inverse probability weight is \(1/e(L)\) for a treated person and \(1/\{1-e(L)\}\) for an untreated person. The key population property is that, when these weights use the true propensity score, the weighted covariate distribution in either treatment group reproduces the marginal covariate distribution.
Derivation. Conditional on \(L\), \(E[A\mid L]=e(L)\). Therefore, using conditional expectation, $$ E\left[\frac{A h(L)}{e(L)}\right] =E\left[E\left\{\left.\frac{A h(L)}{e(L)}\right|L\right\}\right] =E\left[\frac{h(L)E(A\mid L)}{e(L)}\right] =E[h(L)]. $$ The denominators are positive by the stated positivity condition, and integrability ensures these expectations exist. For controls, \(E[1-A\mid L]=1-e(L)\), so the same calculation gives $$ E\left[\frac{(1-A)h(L)}{1-e(L)}\right] =E\left[\frac{h(L)\{1-e(L)\}}{1-e(L)}\right] =E[h(L)]. $$ Taking \(h(L)=1\) shows that each arm’s expected total weight is one. Taking \(h(L)\) to be an indicator of a covariate-defined set shows that the weighted covariate distribution in that arm matches the population distribution. The identities concern population expectations; with finite data, balance is generally approximate. \(\square\)
If a fitted model uses \(q(L)\) instead, the treated weighted expectation is \(E[A h(L)/q(L)]=E[e(L)h(L)/q(L)]\), when this expectation exists. It equals \(E[h(L)]\) in general only when the probability values are right in the relevant way. A high AUC does not make the ratio \(e(L)/q(L)\) equal to one.
Worked Example: Same High AUC, Very Different Weighted Balance
Let \(L\) have two equally common values, \(0\) and \(1\). Suppose the true treatment probabilities are \(e(0)=0.05\) and \(e(1)=0.95\). A fitted model instead reports \(q(0)=0.01\) and \(q(1)=0.99\). Calculate the AUC and the covariate distribution after weighting by \(q\).
AUC. Among treated people, the proportion with \(L=1\) is \(0.5(0.95)/\{0.5(0.95)+0.5(0.05)\}=0.95\). Among untreated people, the proportion with \(L=1\) is \(0.5(0.05)/\{0.5(0.05)+0.5(0.95)\}=0.05\). A treated person’s score is higher than an untreated person’s score only when the treated person has \(L=1\) and the untreated person has \(L=0\), which has probability \(0.95(0.95)=0.9025\). The tie probability is \(0.95(0.05)+0.05(0.95)=0.095\). Thus $$ \operatorname{AUC}=0.9025+\frac{1}{2}(0.095)=0.95. $$
Weighted treated group. The population mass of treated people with \(L=1\), weighted by \(1/q(1)\), is \(0.5(0.95)/0.99=0.4798\) (approximately). The corresponding mass with \(L=0\) is \(0.5(0.05)/0.01=2.5\). The weighted proportion with \(L=1\) is therefore \(0.4798/(0.4798+2.5)\approx 0.161\), rather than the population proportion \(0.5\).
Weighted untreated group. The weighted mass with \(L=1\) is \(0.5(0.05)/(1-0.99)=2.5\); the mass with \(L=0\) is \(0.5(0.95)/(1-0.01)\approx0.4798\). The weighted proportion with \(L=1\) is \(2.5/(2.5+0.4798)\approx0.839\). The two weighted groups are far apart, despite an AUC of \(0.95\). The fitted scores got the ranking right but exaggerated the probabilities’ distance from one-half.
For comparison, the correct probabilities \(0.05\) and \(0.95\) have exactly the same ranking and AUC, but the weight identity shows that they recover a population proportion of \(0.5\) in both arms. This example separates discrimination from probability accuracy and weighted balance.
Perfect discrimination can signal no overlap
High discrimination is not automatically a sign of model failure: a correctly specified model can discriminate well. But exceptionally high discrimination can arise when treatment is nearly determined by covariates. That pattern should prompt careful overlap and weight checks, because it can accompany severe practical positivity problems or, in the limiting case, a structural positivity violation.
Worked Example: AUC of One and Undefined Weights
Again let \(P(L=0)=P(L=1)=0.5\), but now suppose \(e(0)=0\) and \(e(1)=1\). The score \(s(L)=L\) assigns a higher score to every treated person than to every untreated person. There are no cross-group ties, so \(P\{s(L_1)>s(L_0)\}=1\) and the AUC is one.
Yet \(L=0\) people are never treated and \(L=1\) people are never untreated. The treated weight \(1/e(L)\) is undefined for \(L=0\), and the control weight \(1/\{1-e(L)\}\) is undefined for \(L=1\). More fundamentally, the data contain no treatment comparison within either covariate stratum. No improvement in ranking can create the missing comparisons; the positivity condition fails.
This limiting example is different from the first example, where all true probabilities were strictly between zero and one. There, weights existed but weighting was unstable and the misspecified probabilities produced poor balance. Here, weights cannot be defined for the full target population using these formulas.
Chance discrimination can accompany ideal weights
The reverse implication also fails: a low AUC does not imply bad weights. If treatment is randomized with a constant probability, the propensity model has no covariate-based ranking to perform. Its AUC is at chance level, while its weights are constant and covariate balance holds in the population.
Worked Example: AUC of One-Half With Constant Correct Weights
Let \(L\) be binary with \(P(L=1)=0.30\), and let treatment be randomized independently of \(L\) with \(P(A=1\mid L)=0.5\). The correct score is \(e(L)=0.5\) for everyone. Every treated-control score comparison is a tie, so the AUC is \(0+\frac12(1)=0.5\).
Every treated person receives weight \(1/0.5=2\), and every untreated person also receives weight \(1/(1-0.5)=2\). Before weighting, the treated mass with \(L=1\) is \(0.30(0.5)=0.15\), and the treated mass with \(L=0\) is \(0.70(0.5)=0.35\). Multiplying both by 2 gives \(0.30\) and \(0.70\), which normalize to a proportion \(0.30\) with \(L=1\). The same calculation applies to controls. Thus the AUC is \(0.5\), yet the correct weights preserve the population covariate distribution in each arm.
In a finite randomized sample, the observed covariate proportions can differ between arms by chance. Constant weights cannot remove random sample imbalance, but the population identity remains true, and the model has not failed simply because its AUC is low.
What to inspect alongside AUC
AUC answers a narrow ranking question. It does not verify probability calibration, adequate overlap, covariate balance after weighting, or causal identification. Calibration plots for treatment probability models assess a different feature: whether predicted probabilities agree with observed treatment frequencies across relevant ranges. Neither calibration nor AUC alone establishes that the resulting weights are suitable for the target analysis.
Use the diagnostic that matches the concern. The earlier material on covariate balance as a weight diagnostic and propensity score overlap plots addresses properties that an ROC summary cannot replace. In practice, inspect the following together:
- Probability calibration: compare predicted probabilities with observed treatment frequencies. A model can rank well but be systematically too extreme or too moderate.
- Overlap and support: check whether both treatment groups occur across the covariate regions needed for the target comparison. Very high discrimination can be a warning to investigate, not a proof of positivity failure.
- Weights: examine their range and concentration. Probabilities near zero for treated people or near one for untreated people can produce large weights.
- Post-weighting balance: compare covariate distributions after weighting, not only the treatment model’s ability to classify treatment.
| Diagnostic | Question it addresses | What it does not establish alone |
|---|---|---|
| AUC or ROC curve | Does the score rank treated people above untreated people? | Accurate probabilities, overlap, or weighted balance |
| Calibration assessment | Do predicted probabilities correspond to observed treatment frequencies? | All relevant covariate balance or causal identification |
| Overlap and support checks | Are treatment comparisons available across the target covariate distribution? | Correct model probabilities or adequate balance by themselves |
| Weight and balance checks | Are weights concentrated, and do measured covariates balance after weighting? | Unmeasured confounding control or correctness of every causal assumption |
Common mistake and practical interpretation
Wrong reasoning: “The AUC is \(0.95\), so the propensity model is excellent and its inverse probability weights are trustworthy.” This conclusion confuses accurate ordering with accurate probability values. In the first worked example, the score’s AUC is \(0.95\), but the misspecified probabilities produce weighted proportions of about \(0.161\) and \(0.839\) for \(L=1\) in the two arms.
Correction: Treat AUC as a ranking summary, not a pass/fail test for weighting. Assess probabilities, overlap, weights, and post-weighting balance directly. Also avoid the opposite mistake: a high AUC alone does not prove weights are invalid. A genuinely correct propensity model may rank well, but extreme treatment predictability can make the weights unstable even when the probabilities are correct.
These diagnostics have limits. Balance is assessed only for measured covariates and summaries that are checked; passing it does not prove absence of unmeasured confounding. Calibration is also usually assessed through finite samples and chosen groupings, so apparent agreement cannot guarantee the true conditional probabilities. Finally, no discrimination statistic can repair a lack of support: when a treatment option is absent for some covariate patterns, the observed data do not supply that comparison.
Check Your Understanding
Try each question, then compare your reasoning with the solutions.
- Why does replacing a propensity score \(s(L)\) by a strictly increasing transformation \(g\{s(L)\}\) leave its AUC unchanged?
- A fitted model has AUC \(0.90\). Does this show that its inverse probability weights yield covariate balance? Explain.
- In a population with two equally common covariate groups, the true treatment probabilities are \(0.10\) and \(0.90\). A model reports \(0.20\) and \(0.80\), in the same group order. Can the AUC distinguish the fitted scores from the true scores? What should be checked instead?
- What does AUC equal when every subject receives the same propensity score, and can the corresponding weights still be valid?
Solutions. (1) Strict increase preserves every pairwise ordering and every tie; therefore both probabilities in the AUC definition stay unchanged. (2) No. AUC assesses ranking, not the probability values used in the weight denominators. Check calibration, overlap, the weights themselves, and post-weighting balance. (3) No, since the two sets of scores have identical ordering and no ties between the groups, so their AUCs agree. Check whether the probabilities are accurate and whether weighting achieves balance with acceptable weight behavior. (4) If all scores tie, AUC is \(0+\frac12(1)=0.5\). Yes: under randomized treatment with a constant true probability, inverse weights are constant and recover population covariate distributions in expectation, as shown in the worked example.