Comparing Tipping Points Across Analyses

Why Comparisons Need Alignment

A tipping point is not a stand-alone property of a dataset. It depends on the analysis being examined, the sensitivity scenario being varied, and the rule used to decide whether the conclusion has changed. Consequently, two reported thresholds can look directly comparable while answering different questions. For example, one might mark the loss of statistical significance and another a reversal in effect direction; one might measure a shift in outcome points and another a shift in standard deviations.

This tutorial concerns comparing thresholds from prespecified alternative analyses, such as analyses using different populations or models. The goal is not to decide which analysis is correct from its threshold alone. Rather, it is to describe how much the conclusion changes across analyses under comparable sensitivity scenarios—and to state clearly when a comparison is not on a common basis.

The definition includes the decision rule and the path, not merely a number. A threshold for losing a confidence-interval conclusion is different from one for reversing the estimated effect direction. If there are multiple crossings, a report should specify whether it compares first crossings or another named boundary. The first-crossing convention is useful for simple one-directional paths, but does not replace inspection of the full path when the decision can change more than once.

What Must Be Aligned

Before comparing thresholds, check three features. First, use the same decision rule, such as whether a confidence interval excludes zero in the beneficial direction. Second, express the sensitivity scenarios on the same scale, or explicitly convert them to a common scale. Third, check whether the outcome definitions and target contrasts are sufficiently alike for a direct interpretation. The analyses may still be worth comparing when one of these features differs, but the resulting numbers should not be described as a direct ranking.

The comparison is most straightforward when the analyses use the same outcome, contrast, decision rule, and sensitivity scale, and differ only in a feature of interest such as the analysis population. When they do not, alignment may require a defensible conversion. A conversion should preserve the substantive meaning of the scenario, not just make the numbers look alike.

Deriving a Comparable Threshold

A simple sensitivity model makes the comparison explicit. Suppose analysis \(a\) estimates a beneficial effect, and under sensitivity value \(\delta\) its estimate is \(\hat{\theta}_a(\delta)=\hat{\theta}_{a0}+b_a\delta\), where \(\hat{\theta}_{a0}\) is the estimate at the reference scenario and \(b_a\) is the sensitivity slope. Suppose its standard error is treated as fixed along this path, and the conclusion rule is that the lower endpoint of a two-sided 95% normal confidence interval remains above zero. This rule is satisfied when \(\hat{\theta}_a(\delta)-1.96SE_a>0\).

If \(b_a<0\), increasing \(\delta\) reduces the estimated benefit. Solving the boundary equation gives the threshold:

provided the starting lower confidence limit is positive and the path reaches this boundary. This calculation assumes the stated linear estimate path, fixed standard error, and normal critical value; a nonlinear model or changing standard error requires finding the crossing from the corresponding decision function instead. The derivation also shows why identical effect estimates need not yield identical thresholds: standard errors and sensitivity slopes affect the boundary. A larger threshold is not automatically evidence of a better or more credible analysis; its interpretation depends on what one unit of \(\delta\) represents.

Converting Sensitivity Scales

Analyses sometimes encode a similar scenario in different units. For example, one model may vary a missing-outcome shift in endpoint points, while another reports that shift in standard-deviation units. A comparison requires a conversion based on a stated reference quantity. If one reference standard deviation is 12 endpoint points, a threshold of 0.70 standard deviations corresponds to \(0.70(12)=8.4\) points. That conversion is meaningful only if the standard deviation is the intended common reference and the two parameters describe the same kind of shift.

Outcome Definitions Can Block a Direct Ranking

Two analyses may use the same clinical data yet define different outcomes—for example, a mean change in a symptom score and the proportion of participants who meet a responder threshold. Even if both analyses use the rule “the 95% confidence interval must exclude zero in the beneficial direction,” their tipping thresholds are not automatically comparable. A shift of one score point is not a shift of one percentage point in the responder rate. The relationship depends on the outcome distribution and on how the responder definition cuts that distribution.

Diagnostics, Alternatives, and Limitations

A comparison should be checked against the actual decision paths, not just the final threshold values. For each analysis, verify that the reference scenario satisfies the stated rule, that the reported threshold is on the relevant path, and that the decision changes at the claimed boundary. If the decision is non-monotone, there may be more than one crossing; comparing only the first crossing answers a different question from comparing the entire set of scenarios that preserve the conclusion. The earlier material on multiple tipping points and two-parameter boundaries addresses those shapes; a single-number comparison should not conceal them.

• Different decision rules: A threshold for loss of statistical significance cannot be ranked directly against a threshold for effect-direction reversal. Recalculate or report both analyses under a common rule when that comparison is scientifically relevant.
• Unmatched units: A threshold in points and one in standard deviations are not directly comparable. Convert only using a defined reference and a shared substantive interpretation.
• Different reference scenarios: Threshold distances from different baselines can mislead. State each baseline and, when possible, evaluate the analyses from a common reference scenario.
• Changing uncertainty: The illustrative derivation treated the standard error as fixed. If it changes with the sensitivity parameter, calculate the confidence interval at each scenario and locate the crossing from the full decision function.
• Different populations or outcomes: A threshold difference describes how the analyses respond to the specified sensitivity path; it does not by itself identify which population or outcome definition should be preferred.

The main alternative to comparing one-dimensional thresholds is to compare the decision function across a broader range of scenarios, especially when there are multiple sensitivity parameters or repeated crossings. A single threshold is concise when one path and one crossing answer the question, but it suppresses information about other paths and regions. Whatever summary is chosen, the sensitivity scenario should remain substantively interpretable.

Finally, a tipping threshold is a sensitivity result conditional on the analysis and assumptions used to construct it. It is not a probability that the analysis is wrong, a measure of sampling uncertainty in the threshold by itself, or a test that selects a preferred model. Threshold uncertainty and broader regions of robustness require separate treatment. Comparisons are most useful when they make the conditional nature of the result explicit.