Common Error: Treating Nonsignificance As Robustness

Why Repeated Nonsignificance Can Mislead

In sensitivity analysis, an estimated treatment effect depends on assumptions about the data and the missing outcomes. It is tempting to call the result “robust” if a test fails to reject a zero-effect null under every scenario considered. But that pattern says only that the analyses did not provide sufficient evidence against zero at the chosen significance level. It does not show that the effect is small, that clinically important benefit or harm has been ruled out, or that the treatment-effect conclusion is stable in a clinically meaningful sense.

The distinction is between not detecting an effect and establishing that effects large enough to matter are implausible. Confidence intervals and clinical thresholds make this distinction visible. The earlier material on Clinical Benefit Thresholds Under Sensitivity Analysis established how to define a clinically important effect; here the focus is on why nonsignificance alone cannot establish robustness relative to such a threshold.

The numerical threshold must be meaningful for the outcome scale and chosen before interpreting results. If the scale or clinical question calls for different or asymmetric thresholds, use those instead. “Negligible” is not a synonym for “not statistically significant”: it is a claim about the size of effects relative to stated boundaries.

What a Nonsignificant Test Does—and Does Not—Say

Consider a two-sided test of \(H_0:\theta=0\) against \(H_a:\theta\ne0\), using a normal approximation. Let \(\hat{\theta}\) be the estimate, \(SE\) its standard error, and \(\alpha=0.05\). Under the usual Wald procedure, the test statistic is \(z=\hat{\theta}/SE\), and the 95% confidence interval is \(\hat{\theta}\pm1.96SE\). The test fails to reject when \(|z|\le1.96\), which is equivalent to the corresponding 95% confidence interval including zero.

That equivalence concerns the null value zero only. It says nothing by itself about whether the interval includes clinically important effects. An interval can include zero and also extend beyond a clinical threshold. This often happens when the data are imprecise: the estimate may be close to zero, but the range of effects compatible with the data is wide.

The same logic applies separately to each sensitivity scenario, provided the test and interval are valid for that scenario. If every interval includes zero, each analysis is individually nonsignificant under this procedure. That does not imply that every interval lies inside the negligible range. Nor does it imply that the scenarios give the same estimate, the same uncertainty, or the same clinically relevant conclusion.

Assessing Effects Against Clinical Boundaries

A threshold-based assessment asks a different question from a point-null test. Rather than asking whether the data rule out exactly zero, it asks whether the uncertainty range rules out effects beyond the clinically important boundaries. For a confidence-interval approach, compare both endpoints with both boundaries. If the full interval lies inside \((-5,5)\), then the values covered by that interval are all within the defined negligible range. If it crosses or reaches a boundary, the interval has not ruled out effects at or beyond that boundary.

For example, an interval from \(-2\) to \(4\) points includes zero but is wholly within \((-5,5)\). It can therefore be compatible with a negligible-effect conclusion, despite a nonsignificant zero-effect test. Conversely, an interval from \(-2\) to \(6\) includes zero and extends into the clinically important benefit region. The same label, “nonsignificant,” would apply to both if each interval includes zero, but their implications for clinical importance differ.

For a formal equivalence test with symmetric margins \((-5,5)\), the standard two one-sided tests procedure at level \(0.05\) corresponds to a 90% confidence interval lying entirely inside those margins. A 95% interval lying entirely inside the margins is more conservative evidence of containment. Whatever procedure is used, its assumptions and confidence level must be stated; do not switch between confidence levels or procedures after seeing which gives the desired conclusion.

Why “Nonsignificant in Every Scenario” Is Not Robustness

Sensitivity scenarios use different assumptions to examine how conclusions may change. A nonsignificant result in every scenario can arise because each estimate is near zero, because each standard error is large, or both. These possibilities have different implications. A result may be precise enough to rule out effects beyond the clinical margins, or so imprecise that it remains compatible with substantial benefit or harm. The p-value alone does not distinguish them.

The following hypothetical scenario table illustrates the issue. Each analysis uses \(SE=2.0\) points, a two-sided 5% Wald test, and a 95% interval. The scenarios are alternative assumptions applied to the same study, not independent replications.

A Short Justification: Why the P-Value Cannot Settle the Clinical Question

Under the Wald procedure, the p-value for a two-sided test is determined by the standardized distance of the estimate from zero, \(|\hat{\theta}/SE|\). Clinical negligibility, by contrast, depends on the positions of the interval endpoints relative to the clinical margins. A p-value therefore cannot, on its own, establish that the interval lies within those margins.

This can also be seen from the confidence interval’s width. With the same estimate, increasing \(SE\) makes the interval wider while making \(|\hat{\theta}/SE|\) smaller. The test then becomes less likely to reject zero, even as more clinically important effects remain compatible with the data. Thus a larger p-value can accompany weaker information about whether the effect is negligible.

This justification assumes that the normal approximation is appropriate and the standard error is valid. If those conditions fail, the test and interval may not have their advertised properties; the basic distinction between a zero-effect test and a clinical-boundary assessment still applies, but inference must use a suitable procedure.

Diagnostics, Alternatives, and Common Pitfalls

When reviewing sensitivity results, inspect estimates, standard errors, intervals, and clinical boundaries together. Do not reduce the analysis to a column of significance labels. Check whether intervals are unusually wide, whether their endpoints cross a clinical boundary, and whether conclusions differ across assumptions. Also verify that every scenario targets the same treatment-effect estimand and uses the same effect scale and clinical thresholds; otherwise, direct comparisons can be misleading.

• Common mistake: “Every p-value exceeds 0.05, so the effect is robustly absent.” The tests have not rejected zero. That does not rule out clinically important effects. Compare the intervals with the clinical boundaries.
• Common mistake: “The point estimates are all smaller than the threshold, so the effect is negligible.” Point estimates do not show the full uncertainty. An interval can extend past a threshold even when its estimate is close to zero.
• Common mistake: “An interval includes zero, so it cannot support negligibility.” An interval can include zero and still lie wholly within the clinical margins. In that case, it may support a negligible-effect conclusion under the stated procedure.
• Common mistake: “Nonsignificant means the scenarios agree.” Different scenario estimates and intervals can all include zero while implying different ranges of plausible clinical effects.

The main alternative to a point-null significance summary is a threshold-based assessment: report how each interval relates to clinically important benefit and harm, and whether it is contained within the negligible range. Equivalence methods provide a formal approach when the scientific claim is that effects lie within prespecified margins. This is not the same as failing to reject a conventional null hypothesis. Neither approach removes uncertainty about the assumptions that define the sensitivity scenarios.

There are important limitations. An interval wholly within a margin supports a conclusion only for the specified estimand, scale, scenario, and interval procedure; it does not validate the missingness assumptions themselves. If results across several scenarios are summarized jointly, their dependence and the intended coverage claim matter. Separate 95% intervals do not automatically constitute a simultaneous 95% confidence statement across all scenarios. Finally, the chosen clinical margins are substantive judgments, not quantities that a nonsignificant test can determine.