Effect Scales and Population Summaries

Why the Effect Scale Matters

A treatment effect is not a single number until two choices are made: which population is being summarized and which effect scale is being used. For a binary outcome, the difference between treatment and control risks is measured in percentage points, while the risk ratio and odds ratio compare risks in relative terms. These summaries answer related but distinct questions. They can differ in magnitude, and their differences across populations do not by themselves establish that the treatment works differently for different people.

The population issue arises because covariate groups can have different baseline risks and different conditional treatment effects. A population with more high-risk individuals can therefore have different average risks and a different effect summary from a population with more low-risk individuals. The scale issue arises because taking a difference, a ratio, or a ratio of odds transforms those risks differently.

Potential-Outcome Risks and Population Summaries

Let \(A\in\{0,1\}\) denote treatment status, \(Y\in\{0,1\}\) a binary outcome, and \(X\) pretreatment covariates. The potential outcome \(Y(a)\) is the outcome that would be observed under treatment level \(a\). For a target population whose covariate distribution is \(Q\), define the conditional potential-outcome risks \(p_a(x)=P(Y(a)=1\mid X=x)\), and the standardized risks \(\mu_a(Q)=\int p_a(x)\,dQ(x)\). Thus \(\mu_a(Q)\) is the average potential-outcome risk if the target population followed treatment level \(a\).

These are causal estimands defined using potential outcomes. To identify them from observed data by standardization, the conditional risks must be identifiable from the data. A common sufficient set of assumptions is consistency, conditional exchangeability \((Y(0),Y(1))\perp A\mid X\), and treatment positivity in the target population: \(0<P(A=a\mid X=x)\) for each \(a\in\{0,1\}\) at \(Q\)-almost every \(x\). Under these assumptions, \(p_a(x)\) can be represented by the observed conditional risk \(P(Y=1\mid A=a,X=x)\). The definitions themselves do not depend on these identification assumptions.

At a covariate value \(x\), the corresponding conditional measures are \(RD(x)=p_1(x)-p_0(x)\), \(RR(x)=p_1(x)/p_0(x)\) when \(p_0(x)>0\), and \(OR(x)\), the ratio of the two conditional odds when both risks are strictly between 0 and 1. A conditional measure describes one covariate profile; a population measure summarizes risks after averaging over \(Q\).

What Aggregates—and What Does Not

The population risk difference is the average conditional risk difference. This follows directly by subtracting the two standardized risks and using linearity of integration:

In a finite set of strata, this is a weighted average of stratum-specific risk differences, with weights given by the target population's stratum proportions. In contrast, a population risk ratio is not generally the ordinary average of the conditional risk ratios. If \(p_0(x)>0\) for \(Q\)-almost every \(x\) and \(\mu_0(Q)>0\), it can be written as a weighted average with weights proportional to the untreated risk:

These conditions matter. If the target assigns positive weight to a covariate value where \(p_0(x)=0\) but \(p_1(x)>0\), \(RR(x)\) is undefined there, and the displayed weighted-average identity cannot represent the positive treatment risk from that stratum. The population \(RR_Q\) may still be defined when \(\mu_0(Q)>0\), but this conditional-ratio formula must not be used. The odds ratio also has no general weighted-average identity of this kind: odds are nonlinear functions of risk, and the ratio of averaged odds is not the average of conditional odds ratios.

The scale also determines what “heterogeneity” means. Equal risk differences across strata do not imply equal risk ratios, and equal risk ratios do not imply equal risk differences. Therefore, a claim that effects vary across groups should specify the scale. Variation on one scale may coexist with constancy on another.

Changing the Target Population

Even when the conditional risks remain fixed, changing \(Q\) changes the standardized risks and can change all three population summaries. This is not a contradiction: the summary describes a different covariate mix. The risk difference is especially transparent because its target value averages the conditional risk differences. For ratio measures, changing the mix also changes the relative contributions of strata with different baseline risks.

Why Odds Ratios Can Change Without Effect Modification on That Scale

Odds ratios are noncollapsible: even if the conditional odds ratio is identical in every covariate stratum, the marginal odds ratio can differ from that common conditional value. Noncollapsibility is a mathematical feature of the odds scale, not necessarily evidence of confounding or a change in the treatment effect across strata. It is important to distinguish this from confounding, which concerns differences in the groups being compared that distort a causal contrast.

Diagnostics, Comparisons, and Limitations

A careful comparison begins by checking that estimates are being compared on the same scale and for clearly defined target populations. If a matching or weighting procedure changes the covariate distribution, an effect estimate for the resulting population need not equal one standardized to the original population. Covariate-distribution diagnostics help clarify which population the analysis represents; they do not, by themselves, establish that the causal assumptions hold.

• Check support and denominators. A risk ratio requires a positive control risk in the target population. The conditional weighted-average identity additionally requires \(p_0(x)>0\) for \(Q\)-almost every \(x\). Odds ratios require nonzero odds in both treatment conditions; risks of exactly 0 or 1 make the corresponding odds boundary values.
• Inspect stratum-specific risks. Very small risks can make relative measures unstable, while a similar absolute change can correspond to very different relative changes at different baselines. Sparse data can also make estimated stratum effects noisy.
• Separate target shifts from conditional changes. If estimates differ after changing the target weights, recompute both standardized risks and the effect measure. A changed summary may result from a changed covariate mix rather than changed conditional risks.
• Do not interpret scale differences as contradictory evidence. A 10-percentage-point difference and a risk ratio of 1.2 answer different questions. Report the scale, risks, units, and target population needed to interpret the result.

Risk differences are often directly useful for absolute impact and are linear under standardization. Risk ratios communicate relative risk, but can be undefined in strata with zero control risk and are sensitive to baseline-risk composition. Odds ratios are convenient in some models, but their marginal values are not collapsible; they should not be read as if they were risk ratios, especially when outcomes are common. No scale is universally best. The choice should reflect the scientific question and should be stated before comparing summaries.

Common Pitfall