What an Effect Estimate Must Say
A number such as “0.70” does not communicate a treatment effect by itself. It could be a risk ratio, an odds ratio, a difference in mean scores, or a model coefficient on a transformed scale. A useful report tells readers which treatment strategies are contrasted, what outcome is measured, what effect measure is used, what the estimate is, and which population it describes. For a numerical outcome, it also gives the outcome scale and units.
This reporting task is separate from estimating the contrast. In an observational study, a causal interpretation depends on assumptions that support identification, such as consistency, conditional exchangeability given measured covariates, and positivity for the target population. Matching or weighting does not, by itself, guarantee those assumptions. State the design and target population clearly enough that readers can understand what the estimate represents.
Let \(A=1\) denote the treatment strategy and \(A=0\) the comparator, and let \(Y(a)\) denote the outcome that would be observed under strategy \(a\). For a target population, the average treatment effect on the outcome-difference scale is \(E[Y(1)-Y(0)]\). The average treatment effect among the treated instead averages that difference over the population that received treatment. These are distinct population-specific estimands; a matching or weighting design may target one rather than the other.
Choose and Name the Outcome Scale
For a binary outcome, such as readmission by 30 days, define the risks \(p_1=P(Y=1\mid\text{treatment strategy }1)\) and \(p_0=P(Y=1\mid\text{comparator }0)\) for the same target population. Three common effect measures answer different questions:
The risk difference is absolute: it describes the change in risk, often reported in percentage points or as events per 1,000 people. The risk ratio is relative: it compares risks as a multiple or percentage change. The odds ratio compares odds, not risks. An odds ratio of 0.70 is not generally a 30% reduction in risk. Its difference from the risk ratio can be substantial when the outcome is common.
For a continuous outcome, a mean difference is expressed in the outcome’s original units. For example, a difference of \(-1.3\) on a pain scale that runs from 0 to 10 is a 1.3-point lower mean score, not a 1.3% reduction. If a transformed outcome or a standardized measure is reported instead, name that scale and explain how to interpret it.
Worked Example: Reporting Absolute and Relative Risks
Suppose a hypothetical weighted analysis targets 1,000 eligible patients after weighting. The estimated 30-day readmission risk is 12% under the treatment strategy and 18% under the comparator. These are weighted, population-level risks, not necessarily raw proportions among the observed patients.
Risk difference: \(p_1-p_0=0.12-0.18=-0.06\). This is a difference of \(-6\) percentage points. On a scale of 1,000 people, \(1{,}000(-0.06)=-60\): the estimated contrast is 60 fewer readmissions per 1,000 people under the treatment strategy.
Risk ratio: \(p_1/p_0=0.12/0.18=0.667\). The estimated risk under treatment is about 0.667 times the comparator risk, or about 33.3% lower relative to that comparator risk, because \((0.667-1)\times100\%\approx-33.3\%\).
The two descriptions are compatible but not interchangeable: one gives an absolute difference and the other a relative comparison. A report could say: “In the weighted target population, the estimated 30-day readmission risks were 12% and 18%; the risk difference was \(-6.0\) percentage points (60 fewer per 1,000), and the risk ratio was 0.667.” The treatment and comparator order must be stated, since reversing them changes the signs or direction of the ratios.
Marginal and Conditional Estimates
A marginal estimate describes an average over a particular population’s covariate distribution. A conditional estimate describes an effect at specified covariate values, or a model-based contrast conditional on included covariates. They need not be numerically equal, even when both are correctly estimated. In particular, odds ratios are non-collapsible: a conditional odds ratio can differ from the marginal odds ratio even without confounding.
For marginal standardization, estimate the outcome mean under each strategy for each covariate profile, then average those predictions over the target population’s covariate distribution. For a binary outcome and discrete covariate profiles \(x\), the identified population risk under strategy \(a\) can be written as follows when consistency, conditional exchangeability, and positivity hold:
Subtracting the two standardized risks gives the marginal risk difference. The derivation is direct: by the law of total expectation, \(E[Y(a)]=\sum_x E[Y(a)\mid X=x]P_{\text{target}}(X=x)\). Under consistency and conditional exchangeability, \(E[Y(a)\mid X=x]=E[Y\mid A=a,X=x]\) where treatment is possible; positivity ensures those treatment-specific quantities are supported in the relevant profiles. Thus averaging the observed conditional outcome means with the target population’s covariate weights yields \(E[Y(a)]\), and taking the difference yields the marginal contrast. If these assumptions or the target weights do not fit the analysis, the calculation does not justify a causal interpretation for that population.
Worked Example: Conditional and Marginal Odds Ratios
Consider a hypothetical target population with two equally common covariate strata. The comparator risks are 0.10 in the first stratum and 0.30 in the second. Suppose a correctly specified illustrative model gives a conditional odds ratio of 0.60 in both strata. To recover each treatment risk from its comparator risk \(p_0\) and odds ratio \(r\), use \(p_1=rp_0/(1-p_0+rp_0)\).
In the first stratum, \(p_1=(0.60)(0.10)/(0.90+(0.60)(0.10))=0.06/0.96=0.0625\). In the second, \(p_1=(0.60)(0.30)/(0.70+(0.60)(0.30))=0.18/0.88=0.2045\). The common conditional odds ratio is 0.60 in each stratum by construction.
For the marginal risks, average over the two equally common strata. The standardized comparator risk is \((0.10+0.30)/2=0.200\), and the standardized treatment risk is \((0.0625+0.2045)/2=0.1335\). The marginal risk difference is \(0.1335-0.200=-0.0665\), or about \(-6.65\) percentage points. The marginal odds ratio is \([0.1335/(1-0.1335)]/[0.200/(1-0.200)]\approx0.616\), not 0.60.
A clear report would identify the model-based conditional odds ratio as conditional, and separately identify the standardized risks and their marginal contrast as applying to this target population. It would not label the conditional odds ratio “the population risk reduction.”
The covariate distribution used for the average matters. The treated population, the matched sample, and a weighted target population can have different distributions. Therefore, two marginal estimates can differ because they target different populations, even if they use the same outcome definition and treatment contrast. Name the target population in the estimate’s description rather than relying on readers to infer it from the method label.
Reporting Continuous Outcomes and Matched-Population Effects
Worked Example: A Mean Difference in the Treated Population
Suppose a hypothetical study uses matching to estimate the average treatment effect among treated patients for hospital length of stay, measured in days. In the matched analysis, the estimated mean under the treatment received by the treated group is 4.2 days, and the estimated counterfactual mean for those same treated patients under the comparator is 5.0 days.
The estimated contrast is \(4.2-5.0=-0.8\) days. A report should say: “Among the treated patients represented by the matched analysis, the estimated mean length of stay under treatment was 4.2 days, compared with an estimated 5.0 days under the comparator; the average treatment effect among the treated was \(-0.8\) days.” The negative sign means a lower estimated mean under treatment because the contrast was defined as treatment minus comparator.
This is not automatically an effect for all eligible patients: the target is the treated population represented by the matching design. Nor does the estimate alone establish that treatment caused shorter stays; that interpretation still depends on the design’s identifying assumptions and the validity of the analysis.
The population label should be as concrete as the design allows. “The matched population” can be insufficient if matching excluded treated patients, used replacement, or changed the distribution of the comparison group. State, for example, that the estimand applies to matched treated participants, to an overlap population, or to a specified weighted population, as appropriate. The earlier material on identifying the population after matching or weighting is relevant here: the target population is part of what the estimate means, not a technical footnote.
Practical Reporting Checks
Before presenting an estimate, check that the reported contrast and its label agree. For a risk difference, treatment minus comparator must be the subtraction shown. For a ratio, state which group is the numerator. For a mean difference, name the outcome scale and units. Give the outcome time window, particularly when risks are cumulative over a specified follow-up period. Distinguish the analysis sample from the population to which the estimand refers.
| Reported quantity | What it communicates | What to name explicitly |
|---|---|---|
| Risk difference | Absolute change in outcome risk | Outcome event, time window, group order, percentage points or events per population size |
| Risk ratio | Relative comparison of risks | Numerator, denominator, and the risks’ target population |
| Odds ratio | Relative comparison of odds | That the measure is an odds ratio, not a risk ratio; whether it is conditional or marginal |
| Mean difference | Absolute difference in average outcomes | Outcome scale, units, group order, and target population |
A concise reporting sentence can follow this structure: “For [target population], the estimated [outcome and time window] under [treatment strategy] was [estimate], compared with [comparator]; the [named effect measure] was [value and units or scale].” If the estimate is conditional, say what it is conditional on. If it is marginal, state the population over which it was averaged. Do not use “adjusted” as a substitute for either description: adjustment alone does not tell readers what population was targeted or what the measure means.
A common mistake is to report “treatment reduced the odds by 30%” when the estimated odds ratio is 0.70. The arithmetic \(1-0.70=0.30\) is a 30% reduction in odds, not necessarily a 30% reduction in risk. The correction is to name the odds ratio and, where useful, also report standardized risks and an absolute risk difference. Another mistake is to report a matched-sample estimate as applying to the original cohort without evidence that the estimand targets that cohort. State the actual target population instead.
Effect measures also have limitations. Relative measures can appear impressive when absolute risks are small; absolute differences depend on the baseline risk and may vary across populations. Odds ratios can be difficult to interpret as risk changes, and conditional and marginal odds ratios can differ. No single scale is universally best: choose a measure aligned with the scientific question, and report enough information for readers to understand its practical meaning. Statistical uncertainty is a separate part of reporting; a point estimate should not be presented as if it were known exactly.
Check Your Understanding
For each question, identify the relevant reporting issue before checking the solution.
- A weighted analysis estimates risks of 8% under treatment and 10% under the comparator. What are the risk difference and risk ratio, and how should each be described?
- A study reports a treatment-minus-comparator mean difference of \(-2.4\) on a 0-to-100 symptom scale. What additional context is needed to interpret it?
- Why might a conditional odds ratio differ from a marginal odds ratio when the conditional odds ratio is the same in every covariate stratum?
- A matching analysis targets the treated patients, but the report calls its estimate the effect “in all eligible patients.” What is the reporting problem?
Solutions. 1. The risk difference is \(0.08-0.10=-0.02\), or \(-2\) percentage points; the risk ratio is \(0.08/0.10=0.80\), meaning the estimated treatment risk is 0.80 times the comparator risk. Name the outcome, follow-up window, treatment order, and target population. 2. State what the symptom scale measures, the units or score interpretation, the target population, and which strategy is subtracted from which. The estimate means a 2.4-point lower mean under treatment, not a 2.4% reduction. 3. The marginal odds ratio is calculated from risks averaged across strata, whereas the conditional odds ratio compares odds within strata. Averaging risks and then converting them to odds does not generally produce the same ratio as comparing conditional odds. 4. The estimate’s target is the treated population represented by the matching analysis, not necessarily all eligible patients. The report should name the matched treated target or justify a different population-specific estimand.