Worked Example: Treatment-Effect Conclusions Across Scenarios

From sensitivity assumptions to a clinical conclusion

A treatment-effect estimate is not a complete conclusion when some trial outcomes are missing. The estimated effect may depend on assumptions about the unobserved outcomes, so a useful analysis shows how the estimate and its uncertainty change under plausible alternatives. This tutorial works through a hypothetical two-arm trial and classifies the conclusions under several such assumptions.

The outcome is improvement in a symptom score at 12 weeks, measured in points; higher values mean greater improvement. The estimand is the intention-to-treat population mean improvement in the treatment arm minus the corresponding mean in the control arm. A positive contrast therefore favors treatment. Suppose clinicians have specified 0.5 points as the threshold for a clinically important benefit. This threshold is a benchmark for interpretation, not a value estimated from the trial.

The shift \(\delta_a\) is measured in outcome points. A negative shift says that, on average, participants with missing outcomes are assumed to have improved less than observed participants in that arm. The shifts are assumptions, not quantities identified by the observed outcomes alone. The introductory material on Pattern-Mixture Models provides the broad framework; here the emphasis is on carrying assumptions through to an effect estimate, interval, and clinical interpretation.

Trial data and a working interval calculation

Consider 100 randomized participants in each arm. In the treatment arm, 80 outcomes are observed, with mean improvement 8.0 points and sample standard deviation 5.0 points. In the control arm, 90 outcomes are observed, with mean improvement 6.0 points and sample standard deviation 4.5 points. Thus, the observed-data missing fractions are \(r_T=20/100=0.20\) and \(r_C=10/100=0.10\).

For the worked calculations, treat each scenario’s shifts and the missing fractions as fixed. Use the observed sample means and standard deviations to quantify sampling uncertainty, with independent arms and a large-sample normal approximation. Under this working approximation, the adjusted arm mean is the observed mean plus a fixed shift contribution, so its estimated standard error is the observed mean’s standard error. The standard error for the treatment-minus-control contrast is therefore

Here \(n_{T,\mathrm{obs}}=80\) and \(n_{C,\mathrm{obs}}=90\). This gives

We use the approximate 95% interval \(\hat{\Delta}\pm1.96SE(\hat{\Delta})\), with margin of error \(1.96(0.7331)=1.437\) points. The interval is conditional on the scenario shifts and this working variance calculation; it does not represent uncertainty about which shifts are appropriate.

How the assumptions change the effect

The adjustment follows directly from the arm-level mixture. If a fraction \(1-r_a\) has observed outcomes and a fraction \(r_a\) has missing outcomes, the assumed full-arm mean is

Subtracting the adjusted control mean from the adjusted treatment mean gives

This derivation shows why the same shift need not have the same effect on the treatment contrast in both arms: the adjustment is multiplied by that arm’s missing fraction. It also makes the direction clear. A more unfavorable assumed shift in the treatment arm reduces the contrast, while a more unfavorable shift in the control arm increases it.

Compare the conclusions, not just the estimates

The calculations can be summarized with separate statements about direction, statistical evidence, and the clinical benchmark. “Not established” does not mean “absent”: an interval may be too wide to distinguish an important benefit from no benefit or harm.

The first scenario supports both a positive effect and an effect exceeding the clinical threshold, conditional on its assumptions. The other two scenarios weaken those conclusions, but neither interval shows that clinically important benefit is impossible. The result is therefore not uniformly robust across these three assumptions. A scenario table should be interpreted as a set of conditional results, not as evidence that one unobserved-outcome assumption is correct.

Assumptions, diagnostics, and limitations

The calculations rely on several assumptions and approximations. Randomization supports comparison of treatment assignments, but it does not determine the outcomes that would have been observed for participants who are missing. Each scenario must therefore state its shifts, including their units, direction, and clinical rationale. The observed means and standard deviations must also be meaningful summaries for the analyzed outcome, and the large-sample normal approximation should be adequate for the observed sample sizes and outcome distributions.

The interval calculation above treats missing fractions and shifts as fixed and uses observed-outcome sampling variability. It does not add uncertainty from estimating shifts, uncertainty about the missingness process, or extra imputation variability. It is a transparent working calculation for the example, not a general claim that this variance formula is appropriate for every fitted sensitivity model. In an actual analysis, the interval procedure must match the estimator and account for relevant sources of uncertainty.

Useful checks include verifying that missing fractions are calculated from the randomized arm totals, that the shift signs match the stated clinical assumptions, and that all adjusted means use the same outcome scale. Recompute the treatment contrast from the two adjusted arm means as well as from the shift formula. If these disagree, the arithmetic or sign convention is wrong. Also check whether the intervals are so wide that the scenario labels (“moderate” or “severe”) create more confidence than the data warrant.

The reference scenario is often described as a missing-at-random-style reference, but it should not be treated as verified simply because it is familiar. Complete-case analysis, model-based imputation, and selection models make different modeling choices; none can identify the missing outcomes without assumptions. Pattern-mixture shifts make one set of those assumptions explicit and allow their consequences to be compared. They do not prove that the chosen scenarios span all plausible departures.

A sound report states the scenario assumptions, the effect and interval in outcome units, and what each interval does and does not support. The scenarios should be selected and summarized according to a prespecified plan where possible; highlighting only the most favorable result can make conditional evidence appear unconditional. Statistical significance and clinical importance are distinct, and conclusions can change when assumptions change even if the observed trial data stay fixed.