What Does a Significance Tipping Point Measure?
A pooled result can be statistically significant under one imputation assumption and not significant under another. A significance tipping point summarizes where that change first occurs along a specified path of sensitivity assumptions. It is not a property of the data alone: it depends on the estimand, the direction and scale of the delta change, and the test used to define significance.
For a scalar estimand, let \(Q(\delta)\) be the pooled estimate after applying sensitivity parameter \(\delta\), and let \(Q_0\) be the null value. For example, \(Q_0=0\) for a null hypothesis of no difference. In multiple imputation, the pooled estimate and its uncertainty are recalculated from the imputations generated under each value of \(\delta\). A common form of the pooled variance is \(T(\delta)=\bar U(\delta)+(1+1/m)B(\delta)\), where \(m\) is the number of imputations, \(\bar U(\delta)\) is the average within-imputation variance, and \(B(\delta)\) is the between-imputation variance. The precise degrees of freedom, denoted \(\nu(\delta)\), depend on the pooling method.
This definition presumes a continuous path or, more generally, that a first boundary can be identified on the values being considered. If the result is already nonsignificant at \(\delta=0\), there is no positive first loss of significance. If it remains significant throughout the prespecified, substantively plausible range, report that no tipping point was found within that range rather than extrapolating beyond it.
Connection to the Pooled Confidence Interval
For a two-sided pooled \(t\) test of \(H_0:Q=Q_0\) against \(H_a:Q\ne Q_0\), suppose the test uses pooled standard error \(SE(\delta)=\sqrt{T(\delta)}\) and degrees of freedom \(\nu(\delta)\). Its matching confidence interval at level \(1-\alpha\) is
The test rejects exactly when this interval excludes \(Q_0\), provided the interval and test use the same pooled estimate, standard error, degrees of freedom, and two-sided testing rule. Thus, for this test, the tipping point can be found by locating the first \(\delta\) for which
The comparison is between the distance from the estimate to the null and the interval's margin of error. If the estimate and margin both vary with \(\delta\), neither can be held fixed without justification. The interval must be recalculated for each sensitivity assumption.
Worked Example: First Loss of Significance and a Later Crossing
Consider hypothetical pooled results for a continuous outcome, expressed in outcome units. Suppose the null is \(Q_0=0\), the two-sided significance level is \(\alpha=0.05\), the pooled degrees of freedom are 20, and the standard error is \(0.3795\) throughout the sensitivity path. The pooled estimate is \(Q(\delta)=1.20-0.50\delta\). For 20 degrees of freedom, take the critical value as \(t_{0.975,20}=2.086\).
The margin of error is \(2.086(0.3795)=0.791637\), or \(0.7916\) outcome units to four significant figures. Therefore the interval is \(1.20-0.50\delta\pm0.7916\). At \(\delta=0\), the interval is approximately \([0.4084,1.9916]\) outcome units, which excludes zero.
As \(\delta\) increases from zero, the estimate moves down toward zero. The first boundary occurs when the estimate equals the positive margin:
At \(\delta=0.8168\), the interval touches zero. This is the first significance tipping point along the increasing-\(\delta\) path. To see what happens farther along, the interval includes zero while the estimate lies between \(-0.7916\) and \(0.7916\). The two boundary values are
Consequently, the interval includes zero for \(\delta\) from approximately \(0.8168\) to \(3.9832\), including the boundaries. At \(\delta=4\), the estimate is \(1.20-0.50(4)=-0.80\) outcome units, and the interval is approximately \([-1.5916,-0.0084]\) outcome units. It again excludes zero, now on the negative side. The tipping point remains the first loss of significance, \(0.8168\); it does not imply that all larger delta values are nonsignificant.
Why the Chosen Test Changes the Threshold
The tipping point is defined relative to a rejection rule. A two-sided test asks whether the estimate is sufficiently far from the null in either direction. A one-sided test asks whether it is sufficiently far in a prespecified direction. At the same \(\alpha\), the one-sided critical value is usually smaller than the two-sided critical value for the same degrees of freedom. When the hypothesized direction is appropriate and specified before examining the sensitivity results, the one-sided test can therefore remain significant farther along a path that moves the estimate toward the null.
For a one-sided test of \(H_0:Q\leq Q_0\) against \(H_a:Q>Q_0\), the matching lower confidence bound is \(Q(\delta)-t_{1-\alpha,\,\nu}SE(\delta)\). Rejection requires this lower bound to exceed \(Q_0\). The relevant tipping point is where that bound first reaches \(Q_0\), not where a two-sided interval first includes it. Changing the tail or the significance level changes the rule and can change the threshold.
Worked Example: Two-Sided and One-Sided Tipping Points
Suppose hypothetical pooled estimates for an outcome in outcome units follow \(Q(\delta)=0.60-0.20\delta\), with \(Q_0=0\), standard error \(0.20\), and 20 degrees of freedom throughout. Compare a two-sided test with \(\alpha=0.05\) to a prespecified one-sided test for a positive effect at the same \(\alpha\). The relevant critical values are \(2.086\) for the two-sided test and \(1.725\) for the one-sided test.
For the two-sided test, the margin is \(2.086(0.20)=0.4172\) outcome units. The estimate first reaches the positive margin at
For the one-sided positive test, the lower-bound margin is \(1.725(0.20)=0.3450\) outcome units. Its tipping point is
In this hypothetical comparison, the one-sided test remains significant to a larger delta value because it uses a less stringent critical value for the prespecified positive direction. These numbers do not justify switching to a one-sided test after seeing the estimates: the test direction and rule must be selected in advance and scientifically defensible.
When Uncertainty Also Changes with Delta
A delta adjustment can change more than the pooled point estimate. It may also change within-imputation variances, between-imputation variance, or the pooled degrees of freedom. In those cases, the tipping point is determined by the changing distance to the null and the changing critical margin together. A straight-line estimate path alone does not guarantee a straight-line significance path.
Worked Example: A Delta-Dependent Standard Error
Consider a hypothetical two-sided test with \(Q_0=0\), fixed critical value 2.00, and pooled results summarized by \(Q(\delta)=1.00-0.40\delta\) and \(SE(\delta)=\sqrt{0.09+0.01\delta^2}\), with both estimate and standard error in outcome units. At \(\delta=0\), the estimate is \(1.00\) and the margin is \(2(0.30)=0.60\), so the interval excludes zero.
The first crossing occurs on the positive side, where \(Q(\delta)=2SE(\delta)\). Squaring both sides and collecting terms gives
The roots are \(\delta=(0.80\pm\sqrt{0.3328})/0.24\), approximately \(0.929\) and \(5.74\). At the first root the estimate is positive, so it is the first boundary encountered: the significance tipping point is approximately \(\delta=0.929\). The second root corresponds to a later crossing on the negative side. For example, at \(\delta=1\), the estimate is \(0.60\) and the margin is \(2\sqrt{0.10}=0.6325\) outcome units, so zero is included. At \(\delta=6\), the estimate is \(-1.40\) and the margin is \(2\sqrt{0.45}=1.3416\), so the interval excludes zero again.
Squaring an equation can introduce roots that do not satisfy its original sign condition. Here both roots do satisfy the corresponding equality in absolute value, but the first-root interpretation still requires checking which boundary is reached first from \(\delta=0\). In an actual multiple-imputation analysis, the critical value may also vary with pooled degrees of freedom, so the interval and its endpoint should be evaluated using the chosen pooling procedure at each delta value.
Assumptions, Diagnostics, and Reporting
A significance tipping point is interpretable only if the analysis defines a coherent comparison across delta values. The estimand, outcome scale, null value, reference groups, and test must remain the same along the path. The delta adjustment must represent the intended departures from the imputation assumptions, and the imputation and analysis procedures must produce valid pooled estimates and uncertainty estimates under each stated scenario. These requirements do not establish that any particular delta is plausible; that judgment comes from the sensitivity assumptions and relevant subject-matter knowledge.
Useful checks include plotting or tabulating the pooled estimate, standard error, degrees of freedom, interval endpoints, and test decision over the examined range. Confirm that the interval and test agree at selected values, especially near the proposed boundary. Check the sign of the estimate as well: a later significant result on the opposite side of the null is not evidence that the original direction has remained robust. If results appear irregular, check for failed imputations, changes in sample or estimand, unstable variance estimates, or inadequate Monte Carlo precision before interpreting the pattern.
A report should state the test, null value, significance level, direction of the delta path, and the range examined, as well as the estimated first tipping point and the outcome units where relevant. It should distinguish a boundary calculated from a continuous model from the nearest value checked on a coarse set of delta values. A tipping point is a sensitivity summary, not a probability that the null is true and not a measure of clinical importance.
Common Mistakes and Alternatives
- Calling every larger delta nonsignificant. This fails when an estimate passes through the null and becomes far enough from it in the opposite direction. Report the first loss of significance and inspect the rest of the path for later crossings.
- Changing the test after seeing the tipping point. One-sided and two-sided tests can yield different thresholds. Select the test and direction on scientific grounds before examining sensitivity results.
- Holding the standard error fixed without checking. Delta can affect pooled uncertainty as well as the point estimate. Recalculate the pooled interval and degrees of freedom under each scenario.
- Treating significance as a measure of importance. Statistical significance concerns a specified test of a null value. It does not by itself establish that an effect is clinically or practically meaningful.
An equivalent alternative is to report the pooled \(p\)-value as a function of delta and identify its first crossing of \(\alpha\). This conveys the same rejection boundary when the \(p\)-value and confidence interval come from the same test. Reporting a sensitivity curve or a table over a substantively justified delta range adds context that a single tipping point cannot provide. The point estimate and interval remain important: two analyses can share a significance threshold while having different effect estimates and uncertainty.
Check Your Understanding
Answer each question using the stated test and delta path. Solutions follow each question.
- A two-sided interval excludes zero at \(\delta=0\), touches zero at \(\delta=1.2\), includes zero until \(\delta=3.0\), and excludes zero again after that. What is the significance tipping point, and what happens after \(\delta=3.0\)?
Solution: The first loss of rejection is at \(\delta=1.2\), so this is the tipping point. After \(\delta=3.0\), the test rejects again; the estimate has become sufficiently far from zero on the opposite side. It would be incorrect to describe every \(\delta\geq1.2\) as nonsignificant. - A two-sided test uses \(Q(\delta)=0.90-0.30\delta\) and a fixed margin of error of \(0.45\) outcome units. Find the first tipping point for \(Q_0=0\).
Solution: The estimate first reaches the positive margin when \(0.90-0.30\delta=0.45\). Thus \(\delta=(0.90-0.45)/0.30=1.50\). This is the first crossing from the positive side. - Why might a one-sided positive test have a different tipping point from a two-sided test at the same \(\alpha\)?
Solution: The rejection rules use different critical values: the one-sided test allocates the rejection probability to the prespecified positive tail, whereas the two-sided test allocates it across both tails. Their matching confidence bounds therefore reach the null at different delta values. The one-sided direction must be chosen in advance. - If delta changes the pooled standard error, is it enough to solve for when the pooled estimate equals a fixed margin calculated at \(\delta=0\)?
Solution: No. The margin, and potentially the pooled degrees of freedom and critical value, may also change with delta. Recalculate the pooled interval under the stated delta scenario and locate its first contact with the null.