Why Sensitivity to Estimand Mismatch Matters
An analysis can be calculated correctly and still answer a different question from the one a trial prespecified. This can happen when the analyzed population has different subgroup weights, when an analysis handles an intercurrent event differently, or when the follow-up summary changes. The resulting estimate may be precise for its own target but not directly answer the intended clinical question.
Sensitivity analysis makes that distinction explicit. Rather than treating the target as if it were unchanged, it describes how the treatment effect and its interpretation vary under stated, plausible alternatives. The aim is not to select whichever answer is most favorable. It is to show what assumptions or target changes would be needed for the substantive conclusion to change.
An estimand specifies what treatment effect is being sought: for example, the population, treatment conditions, outcome, handling of events that occur after randomization, and summary measure. An estimator is the calculation used to estimate that quantity. A test or interval quantifies evidence or uncertainty for a specified target; it does not by itself establish that the target matches the clinical question. A confidence interval for \(\theta_A\) is not automatically an interval for \(\theta_T\).
A useful general notation is \(\theta_T(\eta)\), where \(\eta\) represents assumptions or choices that affect the target. For instance, \(\eta\) might be a population proportion, an assumed difference between observed and counterfactual outcomes after rescue therapy, or weights assigned to follow-up times. The sensitivity analysis reports how \(\theta_T(\eta)\) changes across a justified range of \(\eta\). The range must be scientifically motivated; choosing it after seeing which values preserve a preferred conclusion undermines the analysis.
Population-Weight Sensitivity
Suppose the treatment effect differs across two population strata, such as biomarker-positive and biomarker-negative participants. If the clinical target is the effect in a population with proportion \(p\) in the first stratum, and the stratum-specific effects are \(\tau_1\) and \(\tau_0\), the marginal target is
If the analysis instead combines stratum-specific effects using a different proportion \(q\), its target is \(\tau(q)\). The mismatch is
This identity follows by subtracting the two weighted averages: \([q\tau_1+(1-q)\tau_0]-[p\tau_1+(1-p)\tau_0]\); collecting the \(\tau_1\) terms gives \((q-p)\tau_1\), and collecting the \(\tau_0\) terms gives \(-(q-p)\tau_0\). Thus mismatch is larger when either the weights differ more or the stratum-specific effects are more heterogeneous. If the effects are equal, changing the weights does not change this particular marginal effect.
Worked Example: Enrichment Changes the Population Weights
Hypothetical data. Suppose a trial estimates a treatment benefit of 8 points in biomarker-positive participants and 2 points in biomarker-negative participants. The prespecified target population is 40% positive, but an analysis using the enrolled, enriched sample implicitly gives the positive stratum 75% weight. Assume for this illustration that the stratum-specific estimates are unbiased for the same outcome and follow-up, and set aside their sampling uncertainty.
The target-population effect is \(0.40(8)+0.60(2)=3.2+1.2=4.4\) points of benefit. The analysis-weighted effect is \(0.75(8)+0.25(2)=6.0+0.5=6.5\) points. The mismatch is \(6.5-4.4=2.1\) points.
The identity gives the same result: \((0.75-0.40)(8-2)=0.35(6)=2.1\) points. If 5 points is a prespecified threshold for a clinically important benefit, the target-weighted estimate is below that threshold, whereas the analysis-weighted estimate is above it. This is a difference in the population question being answered, not a correction to the arithmetic.
A sensitivity analysis can vary the target proportion \(p\) while holding the stratum-specific effects fixed. At \(p=0.30\), the effect is \(0.30(8)+0.70(2)=3.8\) points; at \(p=0.50\), it is \(0.50(8)+0.50(2)=5.0\) points. These values show how strongly the conclusion depends on which population is intended. In an actual trial, uncertainty in the stratum-specific effects and in the target population description would also need to be reflected.
Sensitivity to Event-Handling Assumptions
An intercurrent event, such as rescue therapy, can lead to distinct estimands. A treatment-policy estimand compares outcomes as observed after randomization, including outcomes following rescue. A hypothetical estimand instead asks what the comparison would have been under a specified scenario in which rescue did not occur. These answer different questions; the hypothetical target generally requires assumptions about outcomes that were not observed after rescue.
For a simple mean-outcome example, let \(r_a\) be the proportion of participants in randomized arm \(a\) who receive rescue, and let \(\gamma_a\) be the mean difference between their outcome under the hypothetical no-rescue condition and their observed outcome after rescue. The hypothetical mean in arm \(a\) is the observed arm mean plus \(r_a\gamma_a\). Therefore, the hypothetical treatment contrast is
where \(\tau_{TP}\) is the treatment-policy contrast and arm 1 is the experimental treatment. This expression assumes the observed outcomes for participants without rescue are the same under the hypothetical and observed conditions, and that \(\gamma_a\) appropriately represents the mean counterfactual adjustment among rescued participants in arm \(a\). It is a sensitivity model, not a fact identified from the observed outcomes alone.
Worked Example: Varying the Assumed Effect of Rescue
Hypothetical data. An outcome is improvement from baseline, in points, with larger values better. The observed mean improvements are 7.0 points in the experimental arm and 1.8 points in control, so the treatment-policy contrast is \(7.0-1.8=5.2\) points. Rescue occurs in 30% of the experimental arm and 10% of control. Suppose experts judge that, among rescued participants, the mean no-rescue outcome could be 0 to 4 points lower than the observed outcome in the experimental arm, and 0 to 2 points lower in control. Thus \(-4\leq\gamma_1\leq0\) and \(-2\leq\gamma_0\leq0\).
For the illustrative assumptions \(\gamma_1=-2\) and \(\gamma_0=-1\), the hypothetical contrast is \(5.2+0.30(-2)-0.10(-1)=5.2-0.6+0.1=4.7\) points. The hypothetical no-rescue contrast is 0.5 points lower than the treatment-policy contrast under these assumptions.
Across the full stated ranges, the smallest contrast is \(5.2+0.30(-4)-0.10(0)=4.0\) points, and the largest is \(5.2+0.30(0)-0.10(-2)=5.4\) points. Both calculations use the same observed treatment-policy contrast; only the counterfactual assumptions vary. This range does not prove that the hypothetical effect lies between 4.0 and 5.4 points. It shows what follows if the specified ranges and model are accepted.
Follow-Up Summaries Can Also Change the Target
A contrast at a prespecified time point and an average contrast over several visits are not interchangeable summaries. If \(\tau_t\) is the treatment contrast at visit \(t\), a weighted follow-up summary is \(\sum_t w_t\tau_t\), where each \(w_t\) is nonnegative and the weights sum to 1. Changing the weights changes the estimand unless the contrasts are constant over time or the change happens to leave the weighted value unchanged.
Worked Example: Week-12 Effect Versus an Average Across Visits
Hypothetical data. Assume mean treatment contrasts in improvement are 2 points at week 4, 4 points at week 8, and 6 points at week 12. The prespecified target is the week-12 contrast, 6 points. An analysis instead reports an equally weighted average across these three visits. For this calculation, assume that the reported visit-specific contrasts are estimated on the same outcome scale and that there are no missing-data complications.
The analysis summary is \((2+4+6)/3=12/3=4\) points. Its mismatch from the target is \(4-6=-2\) points. More generally, if the weights for weeks 4, 8, and 12 are \(w_4,w_8,w_{12}\), the summary is \(2w_4+4w_8+6w_{12}\), with \(w_4+w_8+w_{12}=1\).
For weights \(0.20,0.30,0.50\), the summary is \(2(0.20)+4(0.30)+6(0.50)=0.4+1.2+3.0=4.6\) points. The difference from the week-12 target is \(-1.4\) points. These summaries describe different features of the effect over time: the week-12 effect versus an average that gives weight to earlier, smaller effects.
How to Conduct and Check a Sensitivity Analysis
A defensible analysis starts by naming the target and the alternative implicit in the actual analysis. It then states which quantities are observed, which are estimated, and which are assumptions. The following sequence helps prevent a numerical sensitivity exercise from obscuring a change in the clinical question.
Write down the intended population, outcome, event strategy, follow-up, and summary. State the treatment contrast and its units.
Determine whether the analysis differs in population composition, event handling, follow-up, or another element that changes the target. Do not describe a target change as merely a different estimator.
Define the weights, counterfactual adjustments, or time-summary weights that connect the analysis to the intended target. Justify plausible values using clinical knowledge or external evidence.
Show the estimated target over the prespecified range, including whether a clinically meaningful threshold or decision changes. Report sampling uncertainty separately where it can be estimated.
Useful diagnostics include checking that population weights sum to 1, confirming that treatment contrasts have consistent direction and units, and verifying that event adjustments are applied to the rescued fraction rather than to the entire arm. For a follow-up summary, verify that weights are nonnegative, sum to 1, and correspond to the intended time period. Plotting the target against a sensitivity parameter can reveal a tipping point: a value at which a specified decision or interpretation changes.
A sensitivity range should include plausible values on both sides of its central assumption when the evidence does not favor one direction. If the conclusion changes within that range, report that dependence plainly. If it does not, the conclusion is robust to the particular mismatch examined, not necessarily to every possible source of bias or target change.
Alternatives, Limitations, and a Common Mistake
Scenario analysis reports results at a small number of explicitly chosen assumptions; a range analysis evaluates a continuum or grid of values. A tipping-point analysis focuses on the assumptions required to change a decision. Partial-identification bounds report a range implied by weak assumptions, often wider but less dependent on a single model. These approaches can complement one another. None substitutes for defining the estimand, and none removes sampling uncertainty unless it is incorporated into the calculations.
Model-based adjustment may produce one estimate for a hypothetical target, but it relies on its model and assumptions. Reweighting can address a population-weight mismatch when the target population and relevant covariates are adequately described and the required population data are available; it does not automatically address event-strategy or follow-up mismatch. A direct estimate of the treatment-policy effect may be preferable when the clinical question concerns outcomes regardless of rescue, but it does not answer a no-rescue question.
Sensitivity analysis has limits. If stratum-specific effects are poorly estimated, a population-weight exercise that treats them as known understates uncertainty. If counterfactual outcomes after rescue are not observed, sensitivity parameters cannot be learned from those outcomes alone without additional assumptions or data. If an outcome is measured on different scales or at incompatible times, a numerical difference may not have a meaningful interpretation. Results are also limited to the ranges and model structures examined.
Check Your Understanding
Answer each question, then compare with the brief solutions.
- Two strata have treatment effects of 5 and 1 points. The target population is 30% in the first stratum, but the analysis weights it at 60%. Calculate the target effect, the analysis-weighted effect, and their difference.
- In a rescue sensitivity model, \(\tau_{TP}=3.0\), \(r_1=0.20\), \(r_0=0.40\), \(\gamma_1=-2\), and \(\gamma_0=-1\). Calculate \(\tau_H\).
- Visit-specific contrasts are 1, 3, and 5 points. What is the equally weighted mean across the visits, and how does it differ from the final-visit target?
- Why does a narrow confidence interval for an analysis-weighted effect not establish that the intended target effect is estimated precisely?
Solutions. (1) Target: \(0.30(5)+0.70(1)=2.2\) points. Analysis: \(0.60(5)+0.40(1)=3.4\) points. Difference: \(3.4-2.2=1.2\) points. (2) \(\tau_H=3.0+0.20(-2)-0.40(-1)=3.0-0.4+0.4=3.0\) points. (3) The average is \((1+3+5)/3=3\) points, which is 2 points below the final-visit target of 5. (4) The interval quantifies sampling uncertainty for the target the analysis estimates; it does not quantify mismatch between that target and the intended one.