Why the Same Trial Can Produce Different Contrasts
Two analyses can use the same participants, randomization, and recorded outcomes yet report different treatment contrasts. That is not automatically a contradiction or an analytical error. A contrast summarizes a particular question: who is being compared, under which treatment conditions, using which outcome and intercurrent-event strategy, and with which population-level summary. Change one of those choices and the target quantity can change.
The useful diagnostic question is therefore not simply “Which answer is correct?” but “Which question does each answer describe?” The earlier material on choosing effect scales distinguishes possible summaries; here the emphasis is on tracing why a change in any estimand component can alter the contrast.
The outcome is part of the question as well as the summary measure. For example, a mean difference in symptom change and a difference in the probability of meeting a response threshold do not summarize the same outcome. Similarly, a treatment-policy strategy incorporates outcomes after an intercurrent event, while a hypothetical strategy targets outcomes under a scenario in which that event did not occur. These targets can differ even if both are motivated by the same trial.
Population Composition Can Change a Contrast
A population-level mean is an average over the people in the target population. If treatment effects vary across subgroups, changing the proportions of those subgroups changes the overall mean difference. This is a direct consequence of averaging subgroup means with the target population’s weights.
Worked Example: Two Target Populations, Two Mean Differences
Suppose a hypothetical randomized trial includes two baseline subgroups. The outcome is improvement in a clinical score, measured in points, and larger values are better. The trial’s subgroup mean improvements are 10 points under treatment \(a\) and 8 under treatment \(b\) in subgroup L; in subgroup H they are 5 and 4 points, respectively. Assume both subgroups are represented in the trial and that these subgroup means are appropriate for each target population considered.
The subgroup differences are \(10-8=2\) points in L and \(5-4=1\) point in H. Consider first a target population that is 80% L and 20% H. Its treatment mean is \(0.8(10)+0.2(5)=8+1=9\) points, and its comparator mean is \(0.8(8)+0.2(4)=6.4+0.8=7.2\) points. The contrast is therefore \(9-7.2=1.8\) points. Equivalently, weighting subgroup differences gives \(0.8(2)+0.2(1)=1.6+0.2=1.8\) points.
Now consider a target population that is 20% L and 80% H. The corresponding means are \(0.2(10)+0.8(5)=2+4=6\) points under treatment and \(0.2(8)+0.8(4)=1.6+3.2=4.8\) points under the comparator. Its contrast is \(6-4.8=1.2\) points, also equal to \(0.2(2)+0.8(1)=1.2\).
The results differ because the subgroup treatment differences are not equal and the target populations use different subgroup weights. The calculations presume that the subgroup mean outcomes apply to the target populations; in an application, that requires suitable representation or justified transport of those means. The two results answer different population questions, rather than showing that the trial data changed.
This weighted-average argument applies to mean differences. Other summaries need not behave as simple weighted averages of subgroup contrasts. In particular, do not infer that a population change must alter every possible treatment contrast, or that a change in a contrast proves subgroup effects differ: the outcome, summary, and assumptions also matter.
Intercurrent-Event Strategies Define Different Outcomes
An intercurrent event can alter what outcome is relevant after randomization. A treatment-policy estimand uses outcomes regardless of whether the event occurs. A hypothetical estimand instead concerns outcomes under a specified scenario in which the event did not occur. Those are distinct targets. Randomization supports comparisons of assigned treatment groups, but it does not reveal an individual’s unobserved outcome under a scenario that did not happen.
Worked Example: Treatment Policy Versus a Hypothetical No-Rescue Strategy
In a hypothetical trial, rescue therapy is an intercurrent event, and the endpoint is a score improvement in points, with larger values better. Four participants are assigned to each arm. The treatment-policy strategy uses each participant’s observed endpoint, including outcomes after rescue. The table also gives assumed no-rescue outcomes for those who received rescue; these are illustrative counterfactual values, not observed data.
| Assigned arm | Observed without rescue | Observed after rescue | Assumed no-rescue outcome for rescued participants |
|---|---|---|---|
| Treatment \(a\) | 8, 10 | 4, 6 | 7, 9 |
| Treatment \(b\) | 6, 8 | 3, 5 | 4, 6 |
For the treatment-policy contrast, the treatment-arm mean is \((8+10+4+6)/4=28/4=7\) points. The comparator mean is \((6+8+3+5)/4=22/4=5.5\) points. The treatment-policy difference is \(7-5.5=1.5\) points.
For the hypothetical strategy, retain the outcomes of participants who did not receive rescue and replace the rescued participants’ outcomes with the stipulated no-rescue values. The treatment-arm mean becomes \((8+10+7+9)/4=34/4=8.5\) points. The comparator mean becomes \((6+8+4+6)/4=24/4=6\) points. The resulting hypothetical contrast is \(8.5-6=2.5\) points.
The difference between 1.5 and 2.5 points follows from changing the intercurrent-event strategy. The hypothetical contrast depends on the assumed no-rescue outcomes; they cannot be checked against the observed outcomes for those same rescued participants. A real analysis would need assumptions and a method that justify estimating the hypothetical target. The numbers here demonstrate the estimand distinction, not an identification method.
A useful check is to ask whether each analysis uses post-event outcomes, replaces them with outcomes under a stated scenario, or excludes a group from the target population. Exclusion is not automatically equivalent to a hypothetical strategy: the remaining participants may differ systematically from those excluded, and the resulting population may no longer be the intended target.
Outcome Definitions and Summary Measures Also Matter
Even when the population and treatment conditions are held fixed, a different outcome definition can produce a different answer. Turning a continuous outcome into a responder indicator, for example, discards information about how far each participant’s result lies above or below the threshold. A different summary of the same outcome can also change the reported scale and numerical value.
Worked Example: Mean Improvement Versus Responder Probability
Consider hypothetical improvement scores, in points, for five participants in each arm. Define a responder as a participant whose improvement is at least 5 points. Assume scores are measured for all ten participants and that the stated threshold is the intended responder definition.
The treatment-arm scores are \(0,0,0,0,10\), giving mean improvement \((0+0+0+0+10)/5=10/5=2\) points. The comparator scores are \(2,2,2,2,2\), giving mean improvement \((2+2+2+2+2)/5=10/5=2\) points. The mean difference is \(2-2=0\) points.
For the responder outcome, one of five treatment participants reaches 5 points, so the treatment proportion is \(1/5=0.20\). None of the five comparator participants reaches 5 points, so its proportion is \(0/5=0\). The risk difference is \(0.20-0=0.20\), or 20 percentage points.
These summaries are not contradictory. The mean difference reflects average improvement, while the responder contrast reflects the proportions crossing a chosen threshold. The threshold and outcome definition must be clinically justified; choosing them after seeing the data can make the contrast misleading.
Worked Example: Two Summaries of the Same Responder Outcome
Suppose a separate hypothetical trial has 60 responders among 100 participants assigned to treatment \(a\), and 40 among 100 assigned to treatment \(b\). Assume the responder definition, target population, and intercurrent-event strategy are identical in both summaries.
The responder proportions are \(60/100=0.60\) and \(40/100=0.40\). The risk difference is \(0.60-0.40=0.20\), or 20 percentage points. The risk ratio is \(0.60/0.40=1.50\). Both summarize the same group proportions, but they use different scales and therefore have different numerical values and interpretations. A value of 1.50 is a ratio, not a difference of 1.50 percentage points.
A Practical Diagnostic and Its Limits
When two reported contrasts differ, compare their definitions before comparing their estimates. The following sequence helps isolate the source without implying that every component can be separated by arithmetic:
Check eligibility, subgroup composition, and whether one analysis standardizes to a different population.
Check the variable, time point, units, and any transformation or responder threshold.
Check whether outcomes after an event are used, replaced under a hypothetical scenario, or otherwise handled.
Check whether the contrast is a mean difference, risk difference, ratio, or another summary, and retain its units.
If the estimand components match, investigate estimation, missing data, model specification, and sampling uncertainty. If they do not match, describe the results as answers to distinct questions.
This diagnostic is not a formula for attributing a total numerical difference uniquely to each component. Components can interact: changing the population can matter more under one outcome definition than another, and different summary measures can respond differently to the same subgroup distribution. A one-component-at-a-time comparison is informative only when the other components are held fixed and the relevant data and assumptions support both targets.
A common mistake is to treat the estimate closest to zero, or the estimate with the narrowest confidence interval, as the uniquely correct treatment effect. That reasoning ignores the target question. The correction is to state which population, outcome, event strategy, and summary each estimate represents, then assess whether that target is relevant and whether its assumptions are defensible. Precision does not make an estimand more appropriate, and numerical agreement does not prove that two analyses target the same quantity.
The main alternatives are not competing calculations of one universal effect. A treatment-policy estimand, a hypothetical estimand, and a while-on-treatment estimand can each be appropriate for a different clinical question. Likewise, a mean contrast and a responder contrast may both be useful while describing different features of the outcome. Their limitations differ: population-specific results may not generalize to another population; hypothetical results depend on assumptions about unobserved outcomes; and threshold-based outcomes depend on a chosen cutoff. The estimand should be selected for the question, not selected because it yields a preferred result.
Check Your Understanding
Answer each question, then compare with the solutions below.
- Two populations contain different proportions of subgroups, but the treatment mean difference is 3 points in every subgroup. Under the weighted-mean setup, will the population mean difference change solely because the subgroup weights change?
- In a trial, one analysis includes observed outcomes after rescue therapy and another targets outcomes under no rescue. Name the component that differs and explain why randomization alone does not reveal the hypothetical outcomes.
- For treatment and comparator, mean improvements are 4 and 3 points. Responder proportions are 0.30 and 0.20. Give the mean difference and risk difference, and explain why they are not competing estimates of one outcome summary.
- A report gives a risk ratio of 1.4 and a risk difference of 0.12 for the same binary outcome. What should be checked before calling the values inconsistent?
Solutions. (1) No. The weighted difference is \(\sum_k w_k(3)=3\sum_k w_k=3\) points for any weights summing to one. (2) The intercurrent-event strategy differs. Randomization assigns treatment; it does not expose rescued participants’ unobserved outcomes under the no-rescue scenario. (3) The mean difference is \(4-3=1\) point; the risk difference is \(0.30-0.20=0.10\), or 10 percentage points. One summarizes a continuous outcome’s mean, the other a binary responder outcome’s probability. (4) Check that the populations, responder definition, event strategy, treatment ordering, and underlying proportions match, and confirm that the measures are reported on their correct scales. A ratio of 1.4 and a difference of 0.12 need not have the same numerical value.