Two Different Questions About Robustness
A sensitivity analysis can leave the clinical interpretation of a treatment effect stable while changing whether a statistical test crosses a significance cutoff. It can also leave statistical significance intact while moving the estimated benefit across a clinically important threshold. These are not contradictions: the two judgments answer different questions.
For example, statistical significance concerns how an estimate compares with its standard error under a specified model and assumption. Clinical interpretation concerns the size of the effect relative to a threshold that represents a meaningful benefit. Because assumptions about missing outcomes can affect the estimate, its standard error, or both, the two judgments need not change together.
The direction convention matters. In this tutorial, positive \(\theta\) means treatment benefit, and larger values mean greater benefit. A point estimate is classified as meeting the threshold when \(\hat{\theta}(a)\geq\Delta_{\mathrm{clin}}\). That is a description of estimated magnitude, not proof that the true effect exceeds the threshold.
For a transparent comparison, the examples use a two-sided Wald test with nominal \(\alpha=0.05\). Under each fixed sensitivity assumption, suppose the estimator is approximately normal with a valid standard error. Then
Here \(\Phi\) is the standard normal cumulative distribution function. The test assesses \(H_0:\theta(a)=0\) against a two-sided alternative. Its p-value is conditional on the chosen assumption and model; it is not the probability that the null hypothesis is true. The 95% Wald interval is likewise conditional on that assumption.
Why the Judgments Can Separate
The test statistic depends on a ratio: estimated effect divided by standard error. A clinical threshold instead compares the estimated effect directly with a fixed amount in clinical units. Consequently, an assumption can change statistical significance by changing precision, even when the estimate remains on the same side of the clinical threshold. Alternatively, the estimate can move across the clinical threshold while remaining many standard errors from zero.
A useful summary therefore keeps at least two columns: whether the estimate meets the clinical threshold and whether the test is statistically significant. When an inferential claim about clinically meaningful benefit is desired, a third question is needed: does the interval provide evidence that the effect exceeds \(\Delta_{\mathrm{clin}}\)? Under the interval convention above, a 95% interval whose lower endpoint is above \(\Delta_{\mathrm{clin}}\) supports that stronger conclusion at the corresponding confidence level. An interval that crosses the threshold does not establish that the true benefit exceeds it, even if the point estimate does.
Worked Example: Clinical Classification Persists but Significance Changes
Consider a hypothetical symptom score measured in points, with positive values representing improvement on treatment relative to control. Suppose the prespecified clinically important benefit is \(\Delta_{\mathrm{clin}}=5\) points. Two plausible sensitivity assumptions give the following estimates and standard errors.
| Assumption | \(\hat{\theta}\) (points) | \(SE\) (points) |
|---|---|---|
| A | 6.0 | 2.5 |
| B | 5.5 | 3.0 |
Under A, \(z=6.0/2.5=2.40\), giving a two-sided p-value of approximately \(0.0164\). The 95% interval is \(6.0\pm1.96(2.5)=6.0\pm4.90\), or \((1.10,10.9)\) points. Under B, \(z=5.5/3.0=1.833\), giving \(p\approx0.0668\). Its interval is \(5.5\pm1.96(3.0)=5.5\pm5.88\), or \((-0.38,11.4)\) points.
Both point estimates exceed the 5-point threshold, so the point-estimate clinical classification is stable across these two assumptions. The statistical-significance classification is not: A rejects \(H_0\) at 0.05, whereas B does not. Neither interval lies wholly above 5 points, so neither scenario alone establishes, at the stated interval level, that the true benefit exceeds the clinical threshold. This example separates a robust point-estimate classification from both statistical significance and stronger inferential evidence of threshold-exceeding benefit.
Worked Example: Significance Persists but Clinical Classification Changes
In a second hypothetical trial, benefit is measured in days without symptoms, and the clinical threshold is \(\Delta_{\mathrm{clin}}=5\) days. Two plausible assumptions yield estimates of 7.0 days with \(SE=2.0\) days and 2.0 days with \(SE=0.5\) days.
For the first assumption, \(z=7.0/2.0=3.50\), so the two-sided p-value is approximately \(0.000465\). The 95% interval is \(7.0\pm1.96(2.0)=(3.08,10.9)\) days. For the second, \(z=2.0/0.5=4.00\), so \(p\approx0.0000633\). The interval is \(2.0\pm1.96(0.5)=(1.02,2.98)\) days.
Both tests are statistically significant at 0.05, so significance is stable across these assumptions. But the point-estimate classification changes: 7.0 days exceeds the 5-day threshold, whereas 2.0 days does not. The first interval includes effects below 5 days, while the second lies entirely below 5 days. Thus, stable evidence against a zero effect does not imply stable evidence of a clinically important effect. In particular, the second scenario is statistically significant because its estimated benefit is precise relative to zero, even though its estimated magnitude is below the clinical threshold.
Reading the Results Without Collapsing Them
The sensitivity set should be defined on substantive grounds, and each estimate and standard error should correspond to the same assumption. Report which assumptions were considered and show the conditional estimate, uncertainty, and clinical threshold in a common unit. Then state separately whether (1) the point estimate meets the clinical threshold, (2) the test rejects no effect, and (3) the interval supports benefit exceeding the threshold.
These statements have different meanings. A clinical point-estimate classification can be robust even when some p-values exceed 0.05. Statistical significance can be robust while the estimated magnitude crosses the threshold. And neither of these summaries alone guarantees that the interval supports a clinically important effect. The clinical threshold should be specified independently of the observed estimates; moving it after seeing the results changes the question being answered.
State which direction means benefit, specify \(\Delta_{\mathrm{clin}}\), and identify the plausible assumptions being compared.
For each \(a\), report \(\hat{\theta}(a)\), \(SE(a)\), the test result at the prespecified \(\alpha\), and a confidence interval using the stated method.
Check whether estimates cross \(\Delta_{\mathrm{clin}}\), whether p-values cross \(\alpha\), and whether intervals extend above the clinical threshold.
Say which conclusion is stable, which changes, and what the conditional analyses do and do not support.
A p-value just above 0.05 is not qualitatively opposite to one just below 0.05. The cutoff creates a binary reporting convention, while the underlying test statistic and p-value vary continuously. For example, a change from \(p=0.049\) to \(p=0.051\) should not be described as a sudden reversal in the clinical evidence. Conversely, a small p-value does not make a small effect clinically important.
Worked Example: Significant Benefit Does Not Establish Threshold Exceedance
Suppose a hypothetical treatment improves a functional score, with positive values denoting benefit and \(\Delta_{\mathrm{clin}}=5\) points. Under one plausible assumption, the estimate is 6.2 points and \(SE=1.3\) points. Assume the conditional Wald approximation and standard error are appropriate.
The test statistic is \(z=6.2/1.3=4.769\), giving a two-sided p-value of approximately \(1.85\times10^{-6}\). The 95% interval is \(6.2\pm1.96(1.3)=6.2\pm2.548\), or \((3.65,8.75)\) points. The estimate is above the 5-point clinical threshold, and the effect is statistically significant relative to zero. However, the interval includes values below 5 points, so this analysis does not establish at the 95% interval level that the true benefit exceeds the clinical threshold.
The appropriate conclusion preserves all three facts: the estimated benefit is clinically important by the point-estimate rule; the null of zero effect is rejected under this assumption; and uncertainty remains about whether the true benefit is greater than 5 points. Calling the result simply “clinically proven” would overstate what the interval shows.
Assumptions, Diagnostics, and Limitations
The comparisons above are meaningful only if each conditional analysis is internally coherent. The sensitivity assumption must define the estimand and analysis being summarized; the estimate and standard error must be calculated under that same assumption; and the standard error must represent the relevant sampling uncertainty. The Wald calculations additionally rely on an approximately normal sampling distribution. If that approximation is poor, a suitable alternative test and interval should be used consistently rather than relying on these formulas.
Several checks help reveal when the interpretation is going wrong:
- Confirm that all estimates, standard errors, and the clinical threshold use the same effect scale, units, and benefit direction.
- Check that the assumed standard error corresponds to the same model and sensitivity assumption as its estimate.
- Compare effect magnitudes and intervals, not only a column of p-values; a threshold comparison cannot be recovered from significance labels alone.
- Look for intervals that cross zero or the clinical threshold and state what each crossing implies.
- Ensure the plausible sensitivity set was justified rather than chosen to preserve a preferred conclusion.
These summaries have limits. Conditional p-values do not average over assumptions or quantify the plausibility of those assumptions. A stable point-estimate classification only establishes stability over the particular set examined, not over every reasonable model. A clinical threshold is also a substantive decision standard: if it is poorly justified or expressed on the wrong scale, the resulting clinical classification will be misleading. Finally, uncertainty about the effect under each assumption and uncertainty about which assumption is appropriate are distinct; a narrow conditional interval does not resolve uncertainty between assumptions.
Common Pitfall
A concise report might say: “Across the specified assumptions, point estimates remained above the 5-point clinical threshold, but the two-sided test of zero effect was significant in only some scenarios. Confidence intervals did not remain wholly above the clinical threshold.” This phrasing makes the stability pattern explicit without treating a p-value cutoff as a clinical decision rule.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A treatment has \(\Delta_{\mathrm{clin}}=4\) units. Across two assumptions its estimates are 5 and 4.5 units, with p-values 0.03 and 0.08. Which judgment is stable under the point-estimate rule, and which is not?
- Explain why two significant p-values do not establish that an effect exceeds the clinical threshold.
- For a benefit estimate of 3 units with \(SE=1\) unit and \(\Delta_{\mathrm{clin}}=4\) units, calculate the Wald \(z\)-statistic and 95% interval. State what they show.
- Why is it incorrect to interpret a p-value of 0.06 as a 6% probability that the treatment has no effect?
Solutions. (1) Both estimates exceed 4 units, so the point-estimate clinical classification is stable. Significance is not stable at \(\alpha=0.05\), because 0.03 is below the cutoff and 0.08 is above it. (2) Significance compares the estimate with zero relative to its standard error; threshold exceedance concerns whether the effect is greater than 4 units. The estimate, interval, and threshold must be considered for that clinical question. (3) \(z=3/1=3.00\), with two-sided \(p\approx0.00270\). The 95% interval is \(3\pm1.96(1)=(1.04,4.96)\) units. The estimate is significant relative to zero and below the clinical threshold, while the interval includes values both below and above 4. (4) The p-value is the probability, under the null model and the stated analysis assumptions, of a test statistic at least as extreme as the observed one. It is not the probability that the null is true.