What a Gamma Tipping Point Measures
A sensitivity analysis can show how a study’s conclusion changes as the allowed degree of hidden bias increases. A Gamma tipping point summarizes that change: it is the smallest value of \(\Gamma\) at which a prespecified decision criterion is met. The criterion might be that a worst-case p-value reaches a chosen level, or that a sensitivity-adjusted effect interval first includes the null.
The tipping point is meaningful only after specifying the outcome test or interval procedure, the null or decision threshold, and the direction of the test if it is one-sided. It is not a direct estimate of how much hidden confounding is present. Instead, it describes how strong the assignment bias would have to be, within the sensitivity model, to change the stated conclusion.
For a test of a null effect, write \(p_\Gamma\) for the worst-case p-value: the largest p-value over assignment probabilities allowed by the sensitivity model at \(\Gamma\). A small \(p_\Gamma\) means the null is rejected even under the most adverse allowed assignment probabilities. If the prespecified significance level is \(\alpha\), the p-value tipping point is the first \(\Gamma\) at which \(p_\Gamma \geq \alpha\). If the study already has \(p_1 \geq \alpha\), the tipping point is 1: the conclusion is not statistically significant even without allowing hidden bias.
Why the Search Moves in One Direction
The sensitivity model at a larger \(\Gamma\) allows all assignment probabilities permitted at a smaller \(\Gamma\), and possibly additional ones. Thus the set over which the worst-case p-value is maximized expands as \(\Gamma\) increases. The maximum cannot decrease when a set expands. Consequently, for a fixed test and null, \(p_\Gamma\) is nondecreasing in \(\Gamma\). This nesting is the reason a threshold can be located by a one-dimensional search.
Here \(\mathcal{P}(\Gamma)\) denotes the assignment distributions allowed at that value of \(\Gamma\). This argument establishes monotonicity of the worst-case p-value, not necessarily smoothness. Exact p-values can change in discrete steps, so a tipping point may be a boundary at which the p-value jumps over \(\alpha\), rather than a value at which it equals \(\alpha\) exactly.
When the sensitivity procedure can be evaluated at any \(\Gamma\), use a grid to bracket the crossing and then refine the bracket by bisection. At each step, evaluate the actual sensitivity procedure; do not assume that the p-value varies linearly with \(\Gamma\). If a calculation is available only on a coarse grid, report the bracket or clearly label any interpolation as approximate.
Worked Example: A Worst-Case P-Value Tipping Point
Suppose a hypothetical matched-pair study has 10 pairs with nonzero outcome differences: in 8 pairs, the treated individual has the higher outcome. Consider the one-sided sign test for a positive treatment effect, with a prespecified threshold \(\alpha=0.10\). Under the sharp null, the worst-case probability that a pair contributes a positive sign is \(q=\Gamma/(1+\Gamma)\). The worst-case p-value is therefore the upper tail probability for a \(\mathrm{Binomial}(10,q)\) random variable.
At \(\Gamma=1\), \(q=1/2\), so
The p-value is below 0.10 when no hidden bias is allowed. To find where it reaches 0.10, use
For example, at \(q=0.55\), the substituted calculation is \(45(0.55)^8(0.45)^2+10(0.55)^9(0.45)+(0.55)^{10}=0.09956\), which is just below 0.10. Numerically solving \(p_\Gamma=0.10\) gives \(q\approx0.5504\). Convert this probability to the corresponding hidden-bias parameter:
Thus the tipping point is approximately \(\Gamma=1.224\). Under this model, allowing a treatment-odds ratio of about 1.224 within a matched pair is enough for the worst-case p-value to reach 0.10. This is a sensitivity statement about the specified sign test and threshold, not evidence that such hidden bias actually exists.
Effect Intervals and the Null-Entry Tipping Point
A confidence interval can be constructed by inverting tests: retain the effect values that are not rejected by the corresponding test. In a sensitivity analysis, the tests are adjusted for each \(\Gamma\), so the resulting interval depends on \(\Gamma\) as well. For a two-sided interval with confidence level \(1-\alpha\), the null effect is included when the sensitivity-adjusted test of that null is not rejected at level \(\alpha\). Under this matched test-and-interval construction, the null-entry tipping point is the same threshold crossing as the p-value tipping point for the null.
As \(\Gamma\) increases, the allowed assignment distributions expand. The sensitivity-adjusted confidence set therefore cannot become more restrictive: effect values excluded at a smaller \(\Gamma\) cannot be made newly excluded solely by allowing more assignment distributions. In particular, an interval that reaches the null is not evidence that the estimated effect has become zero. It means that, at that sensitivity level, the procedure no longer excludes the null value.
Worked Example: Finding When an Interval Reaches the Null
Suppose a hypothetical analysis estimates a treatment effect in days, with positive values representing a longer recovery time in the treated group. Its sensitivity-adjusted 95% intervals are calculated by inverting the same two-sided tests at each \(\Gamma\). The null effect is 0 days. The lower endpoints from the calculation are:
| \(\Gamma\) | Lower endpoint (days) | Does the interval include 0? |
|---|---|---|
| 1.0 | 0.62 | No |
| 1.4 | 0.13 | No |
| 1.5 | 0.02 | No |
| 1.6 | -0.08 | Yes |
At \(\Gamma=1.5\), the lower endpoint remains above 0 days; at \(\Gamma=1.6\), it is below 0. The null-entry tipping point is therefore bracketed by \(1.5 < \Gamma \leq 1.6\). If a single approximate value is useful, linear interpolation between these two computed endpoints gives
This interpolation is only an approximation: interval endpoints need not vary linearly with \(\Gamma\). A more accurate tip requires recalculating the sensitivity interval at intermediate values and narrowing the bracket. Because the interval comes from inversion of the stated two-sided tests, the null-entry criterion is also that the worst-case p-value for a 0-day effect reaches 0.05.
A useful practice is to report both the threshold and its resolution. For example, “the interval first included 0 days between \(\Gamma=1.5\) and \(\Gamma=1.6\), with a linear-interpolation estimate of 1.52” is more transparent than reporting “the tipping point is 1.52” without qualification. If an interval is reported using a method that is not the inversion of the same tests, do not assume that the p-value and interval tipping points coincide.
Practical Search and Diagnostics
A reliable search begins by fixing the decision rule. Specify the null or effect value, whether the test is one- or two-sided, the significance level, and the sensitivity procedure used to calculate the bound. Then evaluate the procedure at \(\Gamma=1\). If the criterion is already met there, no hidden-bias increase is needed. Otherwise, increase \(\Gamma\) until a value satisfying the criterion is found, then refine the crossing.
State the null effect or interval endpoint of interest, the direction of the alternative if relevant, and the threshold \(\alpha\).
Calculate \(p_1\) or the interval at \(\Gamma=1\). This establishes the starting conclusion for the chosen procedure.
Increase \(\Gamma\) until \(p_\Gamma\geq\alpha\), or until the interval includes the null. Keep the last value before and first value at or beyond the crossing.
Use bisection or a finer grid, check that the computed criterion is monotone, and report the method, threshold, and numerical resolution.
Several checks can reveal a calculation or interpretation problem:
- If a worst-case p-value decreases as \(\Gamma\) increases, verify that the same test and null are being evaluated and that the optimization is actually taking the maximum over the allowed assignment probabilities.
- If an inverted sensitivity interval narrows as \(\Gamma\) increases, check the inversion and the definition of the sensitivity set; allowing more assignment mechanisms should not, by itself, add exclusions.
- If a coarse grid appears to skip the threshold, report a bracket rather than claiming that the first displayed grid value is the exact tipping point.
- If changing the direction of the test changes the answer, make sure the direction was specified before examining the sensitivity results. A one-sided tipping point answers a different question from a two-sided one.
- If the sign test is used, confirm that the analysis is based on the stated nonzero paired differences and sharp null. A different outcome test can produce a different tipping point.
How to Interpret and Compare Tipping Points
A p-value tipping point answers, “How far may the assignment odds depart from equality before the chosen worst-case test no longer meets its threshold?” An interval tipping point answers, “At what allowed departure does the confidence set first stop excluding the null?” The first focuses on a testing decision; the second makes the range of effect values explicit. When they use the same test and its inverted interval, their null-crossing thresholds agree.
A fixed-\(\Gamma\) sensitivity analysis gives conclusions over a chosen range of hidden-bias strengths; a tipping-point summary identifies where one particular decision changes. Neither replaces the other. The full curve or a table of \(p_\Gamma\) values shows how quickly the evidence changes, while the tipping point is a concise summary. Comparisons with measured-covariate imbalance can help contextualize a chosen \(\Gamma\), but they do not verify that all unmeasured confounding is no stronger than that benchmark.
The number also depends on the matched design, outcome, test statistic, sidedness, significance level, and treatment of ties or zero differences. Two analyses of the same study can therefore have different tipping points without contradiction if they use different tests or decision rules. A large tipping point is not proof of no hidden confounding, and a small one is not proof that hidden bias caused the result. It describes robustness under a particular model, not the actual strength of an unknown confounder.
Common Mistake
Wrong reasoning: “The tipping point is \(\Gamma=1.5\), so the hidden confounder must increase treatment odds by 50%.” The sensitivity analysis does not estimate a confounder’s actual effect on treatment. \(\Gamma=1.5\) is a bound on a within-set treatment-odds ratio under the specified model; it marks the point at which the chosen statistical conclusion changes. It does not describe a measured or proven covariate.
Correction: State the tipping point conditionally: “Under the Rosenbaum model and this test, the worst-case p-value reached the prespecified threshold at approximately \(\Gamma=1.5\).” Then give the test, threshold, and numerical resolution. This wording separates the model-based robustness result from a claim about what is present in the data-generating process.
Check Your Understanding
Answer each question, then compare with the solutions.
- If \(p_1=0.03\) and \(\alpha=0.05\), has the p-value tipping point been reached at \(\Gamma=1\)?
- Why is the worst-case p-value nondecreasing in \(\Gamma\) under the nested sensitivity model?
- A sensitivity interval has lower endpoint 0.04 at \(\Gamma=1.2\) and -0.06 at \(\Gamma=1.3\), with null 0. What can be concluded without further calculations?
- For an interval obtained by inverting the same two-sided tests, what p-value condition corresponds to the interval including the null?
- What does a tipping point of \(\Gamma=1.8\) not establish?
Solutions. (1) No. The p-value is below 0.05 at \(\Gamma=1\), so the test still rejects at that level; the crossing, if it occurs, is above 1. (2) The allowed assignment distributions at a smaller \(\Gamma\) remain allowed at a larger one. Maximizing the p-value over this larger set cannot produce a smaller maximum. (3) The null-entry tip is bracketed by \(1.2 < \Gamma \leq 1.3\); more calculations are needed to narrow it. (4) The worst-case p-value for the null reaches or exceeds \(\alpha\). (5) It does not establish that an unmeasured confounder exists or that its actual strength is \(\Gamma=1.8\); it is a threshold conditional on the model, test, and decision rule.