Why Treatment Prediction Is Not the Modeling Objective

Prediction and Adjustment Answer Different Questions

A treatment model can classify who received treatment very accurately and still be a poor basis for estimating a treatment effect. Classification asks whether a model can distinguish treated from untreated individuals. Propensity-score adjustment instead uses a score to make the treatment groups comparable on measured pretreatment covariates. These goals are related, but they are not interchangeable: a high accuracy score is not evidence that the covariates are balanced.

Let $A$ indicate treatment, with $A=1$ for treated and $A=0$ for untreated, and let $X$ denote measured pretreatment covariates. The propensity score is the conditional probability of treatment given those covariates. The target is not simply a good prediction of $A$; it is a score that, when used in an adjustment procedure, helps make the covariate distributions comparable across treatment groups.

Definition: The propensity score is $e(X)=P(A=1\mid X)$. A fitted propensity score, $\hat e(X)$, is an estimate of this conditional probability. Treatment-classification accuracy summarizes the proportion of correctly classified treatment labels at a chosen threshold; it does not measure whether adjustment using $\hat e(X)$ achieves covariate balance.

Why Balance Is the Relevant Diagnostic

For a true propensity score, treatment and measured covariates are conditionally independent given the score: within a fixed score value, the distribution of $X$ is the same for treated and untreated individuals. This balancing property explains why the score is useful for adjustment. A brief derivation also shows what classification accuracy does not establish.

\begin{aligned} P(X\in B\mid A=1,e(X)=t) &=\frac{E[\mathbf{1}_{\{X\in B\}}e(X)\mid e(X)=t]}{P(A=1\mid e(X)=t)}\\ &=\frac{t\,P(X\in B\mid e(X)=t)}{t}\\ &=P(X\in B\mid e(X)=t). \end{aligned}

For untreated individuals, the same argument replaces $e(X)$ with $1-e(X)$; when $0<t<1$, the resulting conditional distribution is also $P(X\in B\mid e(X)=t)$. Thus, at a given true score, both treatment groups have the same distribution of measured covariates. A fitted model need not have this property merely because it predicts treatment labels well. The model must capture covariate information relevant to balance, and the adjustment must be assessed in the data where it will be used.

Causal interpretation requires more than balance. In the usual potential-outcomes setup, treatment must be well-defined and consistent, there must be no interference between individuals, treatment assignment must be conditionally exchangeable given the measured pretreatment covariates, and positivity must hold: each relevant covariate profile must have a nonzero probability of either treatment. Balance diagnostics can reveal problems with the adjustment, but cannot verify that all confounders were measured or that exchangeability holds.

Worked Example: High Accuracy but Poor Balance

Suppose a hypothetical data set has 1,000 individuals: 100 treated and 900 untreated. A binary covariate $Z$ is strongly associated with treatment. A treatment classifier predicts $A=1$ when $Z=1$ and $A=0$ when $Z=0$. The observed treatment counts are:

	$Z=1$	$Z=0$	Total
Treated, $A=1$	90	10	100
Untreated, $A=0$	10	890	900
Total	100	900	1,000

The classifier gets 90 treated and 890 untreated individuals correct. Its accuracy is $(90+890)/1{,}000=0.980$, or 98.0%. Its sensitivity is $90/100=0.900$, and its specificity is $890/900=0.9889$. The fitted conditional treatment proportions by $Z$ are $90/100=0.900$ for $Z=1$ and $10/900=0.01111$ for $Z=0$. These figures show strong discrimination.

Now consider another measured pretreatment covariate, $C$, which is associated with the outcome and differs substantially between treatment groups. Suppose the joint counts are as follows: among treated individuals with $Z=1$, 75 have $C=1$ and 15 have $C=0$; among untreated individuals with $Z=1$, 5 have $C=1$ and 5 have $C=0$. For $Z=0$, the corresponding counts are 5 and 5 among treated individuals, and 175 and 715 among untreated individuals. These counts reproduce the treatment-by-$Z$ totals above.

Overall, the proportion with $C=1$ is $80/100=0.800$ among treated individuals and $180/900=0.200$ among untreated individuals. The binary-covariate standardized mean difference (SMD), using the pooled binary standard deviation, is

\text{SMD} = \frac{0.800-0.200} {\sqrt{\{0.800(0.200)+0.200(0.800)\}/2}} = \frac{0.600}{0.400} =1.50.

The score based only on $Z$ has excellent classification accuracy, but it does not balance $C$: the proportions with $C=1$ also differ within each $Z$ group. For example, within $Z=1$, those proportions are $75/90=0.8333$ among treated individuals and $5/10=0.5000$ among untreated individuals. Prediction performance has not established comparability on the outcome-related covariate.

Measure Balance Directly

A balance assessment compares covariate distributions between treatment groups after the proposed adjustment, such as matching or weighting. For a binary covariate, an SMD expresses the difference in proportions relative to a pooled standard deviation. It is a descriptive measure of imbalance, not a hypothesis test or a causal effect estimate. For quantitative covariates, the analogous calculation uses a difference in means divided by a pooled standard deviation.

Definition: For a binary covariate with treated proportion $\hat p_1$ and untreated proportion $\hat p_0$, the pooled-standard-deviation SMD is $$ \text{SMD} = \frac{\hat p_1-\hat p_0} {\sqrt{\{\hat p_1(1-\hat p_1)+\hat p_0(1-\hat p_0)\}/2}}. $$ The sign indicates the direction of the difference; the absolute value describes its magnitude on this standardized scale.

Worked Example: Checking the Arithmetic of a Binary-Covariate SMD

In another hypothetical example, a table reports the distribution of covariate $C$ among 100 treated and 900 untreated individuals. The counts for $C=1$ are $48+18=66$ among the treated and $2+432=434$ among the untreated; the counts for $C=0$ are $32+2=34$ and $18+448=466$, respectively. The denominators are the row totals, 100 and 900.

Treatment group	$C=1$	$C=0$	Total
Treated	66	34	100
Untreated	434	466	900

The proportions with $C=1$ are $\hat p_1=66/100=0.6600$ and $\hat p_0=434/900\approx0.4822$. Substitution gives

\begin{aligned} \text{SMD} &= \frac{0.6600-0.4822} {\sqrt{\{0.6600(0.3400)+0.4822(0.5178)\}/2}}\\ &\approx \frac{0.1778}{0.4869} \approx 0.3651. \end{aligned}

The absolute SMD is about 0.365, indicating a nontrivial difference in this covariate’s prevalence between groups. The row totals matter: using 98 and 902 as denominators would contradict the counts in the table and produce incorrect proportions and an incorrect SMD.

Compare Candidate Models by Their Balance Results

A model’s accuracy can be reported, but it should not be the criterion that decides whether its scores are suitable for adjustment. Instead, apply the intended adjustment and inspect the resulting covariate balance and the availability of comparable treated and untreated individuals. Balance should be checked for all covariates that need to be controlled, not only those emphasized by the fitted treatment model. Marginal balance alone can also miss differences in distributions or joint patterns, so plots and other summaries can supplement SMDs.

Worked Example: Choosing Between Two Candidate Adjustments

Consider a hypothetical study with 100 treated and 100 untreated individuals in a held-out classification sample. Candidate model M1 correctly classifies 92 treated and 88 untreated individuals, for accuracy $(92+88)/200=0.900$. Candidate model M2 correctly classifies 84 treated and 76 untreated individuals, for accuracy $(84+76)/200=0.800$. Suppose the treatment groups have a treated $C=1$ proportion of 0.800, while the untreated proportion after adjustment is 0.450 under M1 and 0.780 under M2.

For M1, the adjusted SMD is

\text{SMD}_{M1} = \frac{0.800-0.450} {\sqrt{\{0.800(0.200)+0.450(0.550)\}/2}} = \frac{0.350}{\sqrt{0.20375}} \approx 0.775.

For M2, it is

\text{SMD}_{M2} = \frac{0.800-0.780} {\sqrt{\{0.800(0.200)+0.780(0.220)\}/2}} = \frac{0.020}{\sqrt{0.1658}} \approx 0.0491.

M1 classifies treatment more accurately, but M2 leaves much less imbalance in $C$ after adjustment. If the goal is covariate balance, M2 is preferable on this diagnostic for $C$. That is not enough to declare M2 adequate: the other relevant covariates, overlap, and causal assumptions still require assessment.

Diagnostics, Alternatives, and Limitations

A practical evaluation separates three questions: does the proposed adjustment balance the measured covariates, is there sufficient overlap to compare treatment groups, and are the causal assumptions plausible? Useful diagnostics include SMDs before and after adjustment, comparisons of covariate distributions, and inspection of the score ranges for treated and untreated individuals. Large residual differences indicate that the proposed adjustment has not achieved its intended balancing goal. Little overlap means some individuals have no credible counterpart in the other group; a classifier may make this problem especially visible by assigning scores near zero or one.

Classification metrics such as accuracy, sensitivity, specificity, and area under the receiver operating characteristic curve describe prediction, not balance. Accuracy can be particularly misleading when treatment is uncommon: predicting “untreated” for everyone can achieve high accuracy while failing to distinguish the treated group at all. Conversely, a lower-accuracy model can still produce better balance for the covariates relevant to adjustment. Neither pattern, by itself, establishes whether a treatment effect is identified.

Outcome regression is another adjustment approach: it models the outcome conditional on treatment and covariates rather than constructing a treatment score for balancing. It has different modeling requirements and diagnostics, and treatment prediction accuracy does not select a suitable model for that approach either. Propensity-score methods offer a way to build comparability on measured covariates, but no score-based adjustment can remove bias from unmeasured confounding. Nor can good observed balance compensate for a violation of positivity or an inappropriate covariate set.

Pitfall: “The treatment classifier is 98% accurate, so the propensity model is adequate.” This confuses prediction of treatment labels with balance after adjustment. Check balance directly for the relevant covariates.
Pitfall: “The SMD is small, so the treatment effect is unbiased.” Balance describes measured covariates in the adjusted sample; it does not prove exchangeability, correct measurement, or adequate overlap.
Pitfall: “A small overall SMD means every aspect of the covariate distributions matches.” A single standardized difference can miss distributional or joint imbalances. Inspect the distributions and the full set of relevant covariates.

Key takeaway: Treatment prediction and propensity-score adjustment have different objectives. Treat accuracy as a description of classification performance; judge an adjustment by the balance and overlap it achieves, while separately assessing the assumptions needed for causal interpretation.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

Why does 95% treatment-classification accuracy not establish that a propensity-score adjustment is adequate?
A binary covariate has proportions 0.60 among treated individuals and 0.40 among untreated individuals. Using the pooled binary standard deviation, calculate its SMD.
In a study with 20 treated and 180 untreated individuals, a classifier predicts every individual as untreated. What is its accuracy, and what important weakness does that accuracy conceal?
Name two diagnostics, other than classification accuracy, that help evaluate a proposed propensity-score adjustment.

Solutions. (1) Accuracy measures correct treatment-label classifications at a threshold; it does not show that covariate distributions are comparable after adjustment. (2) The numerator is $0.60-0.40=0.20$. The denominator is $\sqrt{\{0.60(0.40)+0.40(0.60)\}/2}=\sqrt{0.24}\approx0.4899$, so $\text{SMD}\approx0.4082$. (3) Accuracy is $180/200=0.900$, or 90%. It conceals that the classifier misses all 20 treated individuals, so high accuracy reflects the treatment prevalence and not useful discrimination of treatment. (4) Examples include covariate SMDs after adjustment, comparisons of covariate distributions, and inspection of treated and untreated score ranges for overlap.

	\(Z=1\)	\(Z=0\)	Total
Treated, \(A=1\)	90	10	100
Untreated, \(A=0\)	10	890	900
Total	100	900	1,000