Separating Statistical and Practical Tipping Points

Two Different Questions Along a Delta Path

A delta sensitivity analysis changes an assumption about missing outcomes and examines how the analysis changes in response. Along that path, two distinct conclusions may change. First, does the statistical evidence meet a prespecified significance criterion? Second, does the estimated effect still meet a prespecified practical criterion, such as a minimum clinically important benefit? These are not interchangeable questions: a result can be statistically significant but too small to matter, or practically important in estimated magnitude while too uncertain to be statistically significant.

The introductory material on tipping points establishes the idea of locating a boundary as a sensitivity parameter varies. Here the important refinement is to define which boundary is being located. A delta value that changes a p-value conclusion need not be the delta value that changes a practical decision.

The delta scale has no universal unit or direction. In a particular analysis, it might represent a shift in the imputed outcome mean, but its meaning depends on how the sensitivity model was specified. State that meaning before interpreting any tipping value. At each delta, the imputation and analysis procedure supplies a pooled estimate and uncertainty; conclusions should use the uncertainty calculated for that delta, not an uncertainty value carried over without checking.

For a two-sided test of \(H_0:\theta=0\) at level \(\alpha\), using a normal approximation, statistical significance is determined by

where \(z_{1-\alpha/2}\) is the standard-normal critical value. Equivalently, a two-sided confidence interval \(\hat{\theta}(\delta)\pm z_{1-\alpha/2}SE(\delta)\) excludes zero. If the pooled analysis uses a finite-degrees-of-freedom reference distribution, use its corresponding critical value instead. A practical rule must be stated separately. For example, a benefit-oriented rule might classify an estimated effect as practically important when \(\hat{\theta}(\delta)\geq\theta_{\mathrm{practical}}\). That rule is about estimated magnitude; it is not itself a test of \(H_0\).

Locating and Interpreting the Boundaries

A tipping value is found where its defining rule is at the boundary. For a positive estimated effect and a fixed standard error, the statistical boundary under a two-sided normal test occurs at \(\hat{\theta}(\delta)=z_{1-\alpha/2}SE\). If the estimate decreases as delta increases, the practical boundary for a minimum important benefit occurs at \(\hat{\theta}(\delta)=\theta_{\mathrm{practical}}\). These equations can differ, so the roots need not coincide. In a realistic analysis, both the estimate and standard error may change with delta, and the statistical boundary must then be evaluated using both.

Before interpreting a root, verify that the delta path and the criteria are appropriate: the estimand is the same across scenarios; the imputation and analysis models are applied consistently; and the pooled inference is suitable for each scenario. Also predefine \(\alpha\), the practical threshold, the delta range, and the direction in which the first crossing is sought. The examples below use hypothetical pooled results, a two-sided \(\alpha=0.05\) test, and a normal approximation with \(z_{0.975}=1.96\). The illustrative delta values and standard errors are assumed to be valid outputs of the corresponding delta-adjusted analyses.

Why the Distinction Matters

The examples show that neither ordering is guaranteed. A practical threshold can be crossed while the p-value remains below \(\alpha\), or statistical significance can disappear while the estimate still exceeds a practical threshold. A practical crossing can also occur with no statistical crossing in the chosen range. In all cases, the conclusion is conditional on the delta assumptions and the rules that were specified.

The practical rule itself needs care. A point-estimate rule such as \(\hat{\theta}\geq\theta_{\mathrm{practical}}\) does not account for uncertainty around that estimate. It may be appropriate for a narrowly defined decision convention, but it should not be described as evidence that the true effect exceeds the threshold. A different decision rule could require a confidence interval to lie entirely above the threshold, or could weigh benefits, harms, and costs. Such a rule generally has a different tipping value and must be named and calculated on its own terms. Do not silently switch between rules while comparing delta scenarios.

Similarly, a statistical tipping point depends on the inferential criterion. A two-sided test, a one-sided test, a confidence interval, and a different significance level can yield different boundaries. Report which criterion was used. A p-value is the probability, under the null model and the test assumptions, of obtaining a test statistic at least as extreme as the observed one; it is not the probability that the null hypothesis is true or the probability that the result is practically important.

The calculations above use linear estimate paths and fixed standard errors to make the two boundaries transparent. Delta-adjusted pooled estimates need not be linear, and their standard errors can vary. In that case, solve the statistical boundary using the delta-specific estimate and standard error together; solve the practical boundary using the stated practical rule. Grid searches and root finding can help locate crossings, but a numerical root does not establish that the crossing is unique or substantively plausible.

Diagnostics and Common Failure Modes

Check the results along the full range, not just at the endpoints. A nonlinear path may cross a criterion more than once, and a path may turn back after an initial crossing. If multiple crossings occur, report them or clearly identify the first crossing from the reference delta and the direction of travel. If no boundary occurs inside the assessed range, report that explicitly rather than extrapolating a root beyond it.

• Confirm delta-specific uncertainty. If \(SE(\delta)\) changes, a statistical crossing may result from changing uncertainty as well as a changing estimate. Recalculate the test statistic or interval at each scenario.
• Check the scale and direction. State what a positive or negative delta means in the sensitivity model and give its units when it has units. A numerical tipping value is not interpretable without this information.
• Keep criteria fixed. Changing \(\alpha\), the practical threshold, or the rule for applying that threshold between scenarios makes the resulting boundaries incomparable.
• Check the reference scenario and search range. Say whether a boundary is a loss of significance, a gain of significance, a practical threshold crossing, or no crossing in the investigated range.

A common mistake is to report one “tipping point” without saying whether it refers to a p-value boundary or a practical threshold. For example, the reasoning “the p-value exceeds 0.05 at this delta, so the effect is no longer important” is incorrect: statistical nonsignificance does not show that an effect is practically unimportant. The correction is to evaluate the inferential rule and the practical rule separately, compute their respective boundaries, and describe what changes at each one.

Compared with reporting only the estimate and p-value under one preferred delta scenario, separate tipping values show how sensitive each type of conclusion is to the assumption. Compared with a single decision rule that combines statistical and practical conditions, separate values make clear which condition drives a change. The trade-off is that two boundaries can be harder to summarize, and neither establishes which delta scenario is credible. That assessment requires a defensible sensitivity model and substantive judgment about the missing outcomes.