Tipping Points in a Rare-Disease Trial

Why a Tipping Point Is Useful in a Small Trial

In a rare-disease trial, a modest number of missing outcomes can represent a substantial fraction of the randomized participants. A conclusion based on an analysis under missing at random (MAR) may then depend heavily on how the unobserved outcomes are modeled. A tipping-point analysis makes that dependence explicit: it varies specified departures from MAR and identifies where a prespecified inferential conclusion changes.

The tipping point is not a data-discovered boundary between true and false effects. It is a boundary conditional on the analysis model, the outcome scale, the clinical range chosen for departures, and the inferential rule. Its value is that readers can see whether the conclusion changes under departures judged clinically plausible, and how precisely that boundary is estimated in a small sample.

Consider a hypothetical rare-disease trial with a continuous final outcome measured in points, where higher scores are better. There are 16 randomized participants in each arm, and four final outcomes are missing in each arm. The analysis under MAR estimates a treatment-minus-control mean difference of 3.0 points, with \(SE=1.301\) points and \(df=24\). These are stipulated hypothetical analysis results, not data from an actual trial. We use a two-sided 5% test, equivalently a 95% confidence interval, and treat the supplied standard error and degrees of freedom as the analysis output for the calculations below.

Define the Departures and the Tipping Rule

The equal missing fractions in this example are \(r=4/16=0.25\). The treatment and control departures enter with opposite signs because the contrast is treatment minus control. If missing participants in both arms are assumed to do equally worse than their MAR predictions, then \(\delta_T-\delta_C=0\): the estimated contrast does not change, although the assumed missing outcomes do. It is the difference between departures, weighted by the missing fractions, that changes this contrast.

For the primary calculation, we hold \(SE\) and \(df\) fixed across delta values. This gives a transparent approximate boundary. In an implemented multiple-imputation analysis, delta adjustments can also affect uncertainty; the standard error and degrees of freedom should therefore be recalculated at each grid point when the analysis procedure supports it. The fixed-uncertainty calculation is a teaching illustration, not a claim that uncertainty must remain constant under every imputation model.

Here the rule is whether the two-sided 95% confidence interval for the adjusted contrast excludes zero. With \(df=24\), the required critical value is \(t_{0.975,24}\approx2.0639\). The interval is the adjusted estimate plus or minus \(2.0639SE\). Because the estimate is positive at the boundary of interest, it loses statistical significance when its lower confidence limit reaches zero.

Derive the Tipping Boundary

For an equal missing fraction \(r=0.25\), the adjusted estimate is \(3.0+0.25(\delta_T-\delta_C)\). At the positive-effect tipping boundary, its lower confidence limit equals zero. The following derivation sets that limit to zero and solves for the difference in deltas.

Thus, under this fixed-\(SE\) calculation, the interval just includes zero when the treatment-arm departure is about 1.26 points more negative than the control-arm departure. More negative differences put the lower interval endpoint below zero; the interval then includes zero until the adjusted estimate reaches the negative-significance boundary, beyond which it excludes zero on the negative side. Less negative differences retain a positive lower endpoint. This boundary concerns statistical significance, not the point at which the estimated treatment contrast itself becomes zero. The latter would require \(3.0+0.25(\delta_T-\delta_C)=0\), or \(\delta_T-\delta_C=-12\) points.

Why Small-Sample Uncertainty Moves the Threshold

The threshold depends on both the estimated contrast and its uncertainty. Holding the estimate at 3.0 points and the degrees of freedom at 24 for illustration, increasing the standard error widens the interval. A wider interval can put the MAR analysis itself on the nonsignificant side of the rule; in that situation there is no positive-significance conclusion for a negative departure to overturn. Conversely, a smaller standard error places the significance boundary farther into the adverse-departure region.

Practical Analysis, Diagnostics, and Alternatives

A useful analysis specifies the inferential rule and a clinically defensible range of departures before interpreting where the boundary falls. A grid can show the full pattern rather than a single threshold. For example, varying both arm-specific deltas over ranges elicited from clinicians shows which combinations preserve the conclusion. The equality boundary \(\delta_T-\delta_C\approx-1.2595\) is a straight line in this example because the two missing fractions and the fixed uncertainty are constant. A grid is particularly helpful when the missing fractions differ, since the estimate then changes by \(r_T\delta_T-r_C\delta_C\), not by a common fraction times the simple difference.

Before interpreting a grid, check that the imputation model, outcome scale, delta signs, and treatment contrast are consistently defined. Check that missing fractions match the analysis population and that the standard error and degrees of freedom correspond to the stated model. Plot or tabulate adjusted estimates and interval endpoints, not only a binary significant/nonsignificant label. If the estimated contrast is close to zero, the opposite-significance boundary may also matter; it should not be mistaken for the positive-effect boundary derived here. For multiple imputations, examine convergence, imputed outcome distributions, and whether the uncertainty calculation is stable over the grid. Diagnosing failed fits and unstable imputations is essential: a numerical boundary from an unstable analysis is not reliable evidence of robustness.

The delta approach is a pattern-mixture sensitivity analysis: it states how the distribution of missing outcomes differs from its MAR prediction. It is useful because the departure has a direct outcome-scale interpretation and can be linked to clinical judgments. It does not estimate the deltas from observed outcomes alone. The MAR analysis is the special case with zero departures; reference-based imputation and selection models are alternative sensitivity-modeling strategies, not interchangeable interpretations of the same delta values.

A common mistake is to report “the result is robust because it remains significant until the delta reaches the tipping point” without explaining whether that delta is plausible. The correction is to report the boundary in outcome units, describe its clinical meaning, and compare it with a prespecified or justified range of departures. Another mistake is to interpret a nonsignificant interval as proof of no treatment effect. It means the analysis does not exclude zero at the chosen level; it does not establish equivalence or absence of a clinically important effect. Small samples make both the estimated effect and its uncertainty imprecise, so a sharply stated threshold should not create a false impression of certainty.

The principal limitations are the unverifiable assumptions about missing outcomes, the dependence of the threshold on the chosen inferential rule, and the possibility that a simple additive shift is inadequate. The illustrative fixed-\(SE\) calculation is especially limited if delta changes materially affect imputation variability or degrees of freedom. A full sensitivity analysis should rerun the specified procedure across the grid, recalculate uncertainty, report the assumptions and diagnostics, and show the range of clinically plausible results. The threshold summarizes one feature of that analysis; it does not replace the full sensitivity results.