Why Compare Plausible Estimands?
A clinical trial can support more than one well-defined question about treatment effect. For example, one question may concern the effect of assigning treatment while allowing rescue therapy or treatment discontinuation to occur as they do in practice. Another may ask what the effect would have been if rescue therapy had not been used. These questions can yield different posterior treatment effects without either calculation being an arithmetic error: they describe different treatment strategies.
Sensitivity analysis across plausible estimands makes that distinction visible. It asks how conclusions change when the clinical question changes in a defensible way. It does not, by itself, determine which estimand is most relevant, establish that a hypothetical strategy is credible, or measure uncertainty in the difference between effects. Those require explicit clinical judgment and statistical analysis.
What Must Be Held Clear in a Comparison?
An estimand specifies the treatment effect of interest, including the population, treatment conditions, outcome variable and time point, and how intercurrent events are handled. The treatment-policy and hypothetical strategies are two possible approaches to an intercurrent event. Under a treatment-policy strategy, the outcome is used regardless of whether the event occurred. Under a hypothetical strategy, the target is the outcome that would have been observed under a specified scenario in which the event did not occur.
For illustration, let \(Y^a_{\mathrm{TP}}(t)\) denote the outcome at time \(t\) under assignment to treatment \(a\), with outcomes included regardless of rescue therapy. Let \(Y^a_{\mathrm{H}}(t)\) denote the outcome under assignment to treatment \(a\) in the specified hypothetical scenario that prevents rescue. For a higher-is-better outcome, the corresponding marginal mean treatment effects are:
Here \(a=1\) denotes assignment to the experimental treatment and \(a=0\) denotes assignment to control. Both effects use the same outcome scale, time point, and target population; the intercurrent-event strategy differs. If the outcome is lower-is-better, the sign and clinical interpretation must be adjusted accordingly. A comparison is easiest to interpret when it changes one important estimand component at a time. If population, outcome, time point, and event strategy all change together, the reason for a change in posterior effect is harder to identify.
The treatment-policy estimand generally relies on outcome collection continuing after the intercurrent event, together with a suitable model for the collected outcomes. A hypothetical estimand also needs a defensible way to infer outcomes under a scenario that may not have occurred. For example, it may rely on measured predictors of rescue and outcome, adequate observations across relevant predictor patterns, and a model that correctly represents the outcome under no rescue. Randomization alone does not reveal each participant’s unobserved outcome under a hypothetical no-rescue scenario.
A Practical Comparison and Its Uncertainty
A transparent comparison reports, for each estimand, the posterior mean or another stated summary, a credible interval, and posterior probabilities tied to a clinical decision threshold. If the effects share units and direction, a contrast can also be defined, such as \(\Gamma=\Delta_{\mathrm{H}}-\Delta_{\mathrm{TP}}\). A positive \(\Gamma\), for a higher-is-better outcome, means the hypothetical-strategy effect is more favorable in that posterior draw. It does not automatically mean rescue caused a loss of treatment effect.
To calculate uncertainty in \(\Gamma\), retain the joint posterior dependence between the estimands. For each joint posterior draw, calculate the two effects and subtract them. If the joint posterior is approximately bivariate normal, then:
The covariance term matters because both estimates may use overlapping participants, outcomes, and model parameters. Subtracting posterior means gives the mean contrast, but it does not give its standard deviation or credible interval. Treating the estimands as independent without justification can therefore misstate uncertainty.
Name each estimand’s population, treatment conditions, outcome, time point, and intercurrent-event strategy.
Describe which outcomes were observed and which parts of an estimand rely on modeling or extrapolation.
Report effects with units, uncertainty, and probabilities of exceeding clinically meaningful thresholds.
If a contrast is relevant, compute it from joint posterior draws and interpret it as a contrast between treatment questions.
Worked Example: Treatment-Policy and Hypothetical Effects
Hypothetical trial. A randomized trial evaluates change in a symptom score at 24 weeks, with larger improvement indicating greater benefit. Participants can receive rescue therapy. Outcomes are collected at 24 weeks whether or not rescue was used. A treatment-policy analysis, using all collected outcomes, gives a posterior distribution \(\Delta_{\mathrm{TP}}\mid D\sim N(0.80,0.30^2)\) points. A model for the hypothetical no-rescue strategy gives \(\Delta_{\mathrm{H}}\mid D\sim N(1.20,0.45^2)\) points. Suppose the clinically important threshold is 0.50 points.
Treatment-policy evidence. For a normal posterior \(N(\mu,\sigma^2)\), the probability that the effect exceeds a threshold \(c\) is \(P(\Delta>c\mid D)=\Phi((\mu-c)/\sigma)\), where \(\Phi\) is the standard normal cumulative distribution function. Thus:
The treatment-policy posterior probability that the mean improvement difference exceeds 0.50 points is about 84.1%. Its approximate 95% credible interval is \(0.80\pm1.96(0.30)\), or 0.212 to 1.388 points.
Hypothetical-strategy evidence. Under the model for no rescue:
The approximate 95% credible interval is \(1.20\pm1.96(0.45)\), or 0.318 to 2.082 points. The hypothetical-strategy posterior places more probability above the stated threshold. This comparison is conditional on the model and assumptions used to infer no-rescue outcomes; the larger posterior mean does not establish that the hypothetical scenario is more credible or more clinically relevant.
The posterior mean contrast is \(1.20-0.80=0.40\) points in favor of a larger effect under the hypothetical strategy. Suppose joint posterior draws give \(\operatorname{Cov}(\Delta_{\mathrm{H}},\Delta_{\mathrm{TP}}\mid D)=0.060\) points squared. Then:
Under the approximate normal joint-posterior calculation, \(P(\Gamma>0\mid D)=\Phi(0.40/0.415)\approx\Phi(0.964)\approx0.832\). This is posterior evidence that the hypothetical-strategy effect is larger than the treatment-policy effect, not evidence that the difference is certain or that rescue caused it.
Estimand Sensitivity Versus Assumption Sensitivity
A change in estimand changes the clinical question. A change in model, prior, or missing-outcome assumption can instead change the analysis used to estimate the same question. These forms of sensitivity analysis can both be important, but they should not be conflated. For example, comparing a treatment-policy effect with a hypothetical no-rescue effect changes the event strategy. Comparing two models for the same hypothetical no-rescue effect examines how conclusions depend on assumptions used to estimate that estimand.
Worked Example: Alternative Assumptions for a Hypothetical Strategy
Hypothetical trial. In a 24-week symptom trial, outcomes are collected after rescue therapy, and the analysis targets the same no-rescue hypothetical estimand under two models. The posterior mean improvement difference is 0.62 points with posterior standard deviation 0.30 under a model assuming outcomes are conditionally missing at random given measured history. Under a more conservative model that shifts predicted post-rescue outcomes downward, the posterior mean is 0.28 points with posterior standard deviation 0.31. The clinical threshold is 0.50 points.
Under the first model:
Under the conservative model:
The posterior probability of exceeding the threshold changes substantially, although the estimand remains the same. That change signals sensitivity to the assumptions that supply unobserved no-rescue outcomes. It is not evidence that the two models estimate different clinical questions. A useful report would state both models, explain the clinical rationale for the conservative shift, and show how much data support the required predictions.
Different Outcome Summaries, Different Interpretations
Plausible estimands can also differ in how the outcome is summarized. A mean difference in symptom improvement and a difference in the probability of meeting a responder definition may refer to the same trial and event strategy, but they answer different questions and use different units. Agreement is reassuring only to the extent that both summaries address relevant clinical consequences; disagreement may reveal that a mean effect does not describe changes near a decision threshold.
Worked Example: Mean Improvement and Responder Probability
Hypothetical trial. At 24 weeks, a treatment-policy analysis of symptom-score improvement gives a posterior treatment mean difference of 0.70 points with standard deviation 0.30 points. A responder is defined in advance as an improvement of at least 3 points. A separate treatment-policy analysis gives a posterior treatment-control responder risk difference of 0.12 with standard deviation 0.06. Suppose the clinical thresholds are 0.50 points for the mean difference and 0.10 for the responder risk difference.
For the mean-scale estimand:
For the responder risk-difference estimand:
The first probability concerns a mean difference in symptom points; the second concerns an absolute difference in responder proportions. Neither number is a probability that its estimand is true. Their different values are not contradictory because the estimands and thresholds differ. A clinical interpretation should preserve the units and explain what each threshold means.
Diagnostics, Alternatives, and Limitations
Before interpreting differences, check whether each analysis actually targets its stated estimand. For a treatment-policy strategy, inspect whether outcomes after discontinuation or rescue were collected and how any remaining missing outcomes were modeled. For a hypothetical strategy, inspect how much information supports predictions under the no-event scenario. Sparse observations in important patient-history patterns, implausible extrapolations, poor posterior predictive checks, or large changes under reasonable alternative assumptions are warnings that the hypothetical result is fragile.
Also examine event frequency and timing by treatment arm, the outcome patterns before and after the event, and the extent to which measured predictors explain event occurrence and subsequent outcomes. These checks do not prove the assumptions, but can expose implausible model behavior. Where estimation relies on extrapolation beyond observed experience, present that limitation directly rather than treating the posterior interval as a complete account of uncertainty.
| Approach | Question it addresses | What it does not establish |
|---|---|---|
| One primary estimand with alternative model checks | How robust is inference for the selected treatment question to modeling choices? | Whether a different clinical question would lead to a different conclusion |
| Comparison across plausible estimands | How do posterior treatment effects differ across specified treatment questions? | Which question is most important or whether assumptions for each are credible |
| One estimand with different priors or event assumptions | How sensitive is inference for that question to prior or identification assumptions? | A change in the estimand itself |
These approaches are complementary. A broad set of analyses can become difficult to interpret if every possible definition is presented without a clinical rationale. The main comparison should be limited to estimands that matter to decisions, with any model-sensitivity analyses clearly labeled. Posterior summaries quantify uncertainty conditional on the model and assumptions; they cannot repair an ill-defined question or guarantee that a hypothetical scenario is identified by the available data.
Common Pitfall
Correction: The difference between posterior means compares two estimands. It is not automatically the causal effect of rescue, and it does not express uncertainty about the contrast. To make a statement about the contrast, use joint posterior draws or a justified joint model; to make a causal claim about rescue, a separate causal estimand and assumptions are needed.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- If \(\Delta\mid D\sim N(0.30,0.25^2)\), what is \(P(\Delta>0\mid D)\)?
- If \(\Delta\mid D\sim N(0.55,0.20^2)\), calculate \(P(\Delta>0.40\mid D)\).
- Two estimands have posterior means that differ by 0.40 points. Why is that difference alone not an uncertainty statement or evidence that the hypothetical assumptions are credible?
- Name one diagnostic concern that is especially important for a hypothetical no-rescue estimand.
Solutions. (1) The posterior standard deviation is \(0.25\), so \(P(\Delta>0\mid D)=\Phi(0.30/0.25)=\Phi(1.20)\approx0.885\). (2) Standardize the threshold: \(P(\Delta>0.40\mid D)=\Phi((0.55-0.40)/0.20)=\Phi(0.75)\approx0.773\). (3) The means concern different treatment questions, and their difference alone gives neither uncertainty about the contrast nor evidence that the hypothetical assumptions are credible. The contrast’s uncertainty requires joint posterior information; the assumptions require separate justification and diagnostics. (4) One concern is sparse observed data for patient-history patterns needed to predict outcomes under no rescue. This can make the hypothetical estimate depend heavily on unsupported extrapolation.