Why Confidence-Interval Conclusions Can Have a Tipping Point
A sensitivity analysis can change not only an estimated effect but also the uncertainty attached to it. As a sensitivity parameter varies, a confidence interval may move from entirely on one side of a null or decision threshold to touching or containing that threshold. A confidence-interval tipping point records where this conclusion first changes along a specified path. It is not enough to track the point estimate: the interval’s endpoints determine whether the threshold is included.
The question must be stated precisely. Which confidence level is being used? What threshold is being assessed? Which direction through the sensitivity-parameter space is being examined? And does the interval initially include the threshold or exclude it? These choices determine what “first” means. The calculations below condition on a fixed value of the sensitivity parameter at each point; they do not average over, or assign a probability distribution to, that parameter.
For example, \(c=0\) is often the null value for a difference or other effect measure whose null corresponds to no effect. A decision threshold can instead be a prespecified clinically relevant value. The threshold need not be zero, but it must be expressed in the same units and on the same scale as the interval. In this tutorial, “first” means the smallest parameter value encountered while moving from a stated starting value in a stated direction. It does not mean the smallest value over all possible sensitivity parameters unless that is explicitly the path being studied.
Assumptions and the Endpoint Rule
At every fixed \(\delta\), the interval must be calculated using an inferential procedure appropriate to the analysis under that sensitivity assumption. Its stated confidence level and the uncertainty it is intended to represent must be clear. For instance, if an analysis uses a standard error that accounts for the relevant sampling and imputation uncertainty, that standard error must be recalculated as required when \(\delta\) changes. Reusing a standard error that is not valid for the adjusted analysis can produce a misleading boundary.
For an interval with ordered endpoints \(L(\delta)\leq U(\delta)\), inclusion is equivalent to both endpoint inequalities holding:
A change in inclusion status can occur only when an endpoint meets \(c\), provided the endpoints vary continuously along the path. Thus, candidate boundary values solve \(L(\delta)=c\) or \(U(\delta)=c\). But a candidate root is not automatically a change in conclusion: the other endpoint must also be checked, and the interval’s status on each side of the root must be determined. An endpoint can touch \(c\) and turn back without changing sign; that tangency can still create a point at which the interval includes \(c\).
In the common special case of a symmetric Wald interval, \(\hat{\theta}(\delta)\pm z^*SE(\delta)\), inclusion can also be written as \(|\hat{\theta}(\delta)-c|\leq z^*SE(\delta)\). This equivalence applies to that stated interval construction; it does not license substituting a different standard error or critical value. If the interval is intended to correspond to a two-sided test, its inclusion conclusion is dual to that test’s non-rejection conclusion only when the interval and test use the same model, standard error, critical value, and threshold.
Continuity is useful for locating a boundary between evaluated values, but it does not guarantee a unique boundary or a monotone conclusion. The endpoint functions may cross the threshold several times, or touch it without crossing. A grid can reveal candidate regions, while analytic endpoint equations or careful refinement can locate a boundary more accurately. A sign change is a helpful bracket for a crossing root, not proof that all roots have been found.
Worked Example: First Inclusion of a Null Value
Suppose a hypothetical analysis reports a 95% interval for an effect, measured in points, with \(\hat{\theta}(\delta)=1.20-0.15\delta\), standard error \(0.30\), and normal critical value \(1.96\). Let the null threshold be \(c=0\), and examine \(\delta\geq0\), starting at \(\delta=0\). Assume this interval construction and standard error are valid at each fixed \(\delta\).
The margin of error is \(1.96(0.30)=0.588\) points. Therefore,
At \(\delta=0\), the interval is \([0.612,1.788]\) points, which excludes zero because its lower endpoint is positive. The first possible inclusion occurs when the lower endpoint reaches zero:
At \(\delta=4.08\), the upper endpoint is \(1.788-0.15(4.08)=1.176\) points, so the interval is \([0,1.176]\) and includes zero by the inclusive endpoint rule. For \(\delta\) just below 4.08, \(L(\delta)>0\); just above it, \(L(\delta)<0\), while the upper endpoint remains positive near the boundary. Thus \(4.08\) is the first inclusion boundary on this path. Under the stipulated interval procedure, the interval conclusion changes there from excluding to including the null value.
When the Interval Already Includes the Threshold
A common ambiguity arises when the interval includes \(c\) at the starting value. In that case there is no later “first entry into inclusion” from an initially excluded state. The useful boundary may instead be the first exit from inclusion. Equality still counts as inclusion, so at an exit boundary the interval includes the threshold at the boundary itself and excludes it immediately beyond, assuming the endpoint moves past the threshold.
Worked Example: First Exit from Inclusion
Consider the hypothetical 95% interval \([4+0.2\delta-1.96,\;4+0.2\delta+1.96]\), measured in units of an outcome, for \(\delta\geq0\). Let the decision threshold be \(c=5\). Assume the interval is calculated consistently across this path and that these endpoints describe the intended uncertainty at each fixed \(\delta\).
At \(\delta=0\), the interval is \([2.04,5.96]\), which includes 5. Its lower endpoint increases with \(\delta\), and its upper endpoint is already above 5 at the starting value. The first exit occurs when the lower endpoint reaches 5:
At \(\delta=14.8\), the interval is \([5,8.92]\), so it still includes 5 at equality. For any \(\delta>14.8\), its lower endpoint is greater than 5 and the interval excludes the threshold. Consequently, there is no first entry into inclusion on this path: inclusion holds from the start. The first boundary where the conclusion changes is the exit boundary at \(\delta=14.8\). Solving for the upper endpoint to equal 5 would give a negative \(\delta\), outside the specified path and irrelevant to this first exit.
Locating Boundaries Without Missing Tangencies
For a smooth endpoint, solving an endpoint equation is often direct. When no closed-form solution is available, one can evaluate the endpoint functions over a grid and refine a candidate boundary. The grid-resolution considerations used for tipping boundaries still apply, but a search based only on sign changes has a specific limitation: a root can occur without a sign change. For example, \(L(\delta)-c=(\delta-1)^2\) equals zero at \(\delta=1\) but is positive on both sides. A search that checks only for opposite signs at the ends of a bracket can miss this contact.
At any candidate, verify the full interval status, not merely the endpoint equation. For a possible lower-endpoint contact, calculate \(U(\delta)\) as well as \(L(\delta)\). Then inspect values on both sides, subject to the path domain. If there is no sign change but a near-boundary contact is plausible, inspect the endpoint curve more closely or minimize its distance from \(c\) over the bracket. A coarse grid can miss both a narrow region of inclusion and a tangential contact.
Worked Example: A Tangency That a Sign-Change Search Misses
Consider hypothetical interval endpoints \(L(\delta)=(\delta-1)^2\) and \(U(\delta)=(\delta-1)^2+1\), measured in points, for \(0\leq\delta\leq2\), with threshold \(c=0\). Treat these endpoints as the interval supplied by the stipulated analysis at each fixed \(\delta\).
The interval includes zero exactly when \(L(\delta)\leq0\leq U(\delta)\). Since \(U(\delta)\geq1>0\), the upper-endpoint condition always holds. For the lower endpoint,
At the starting point, the interval is \([1,2]\), so it excludes zero. At \(\delta=1\), it is \([0,1]\), so it includes zero at equality. At \(\delta=0.9\) and \(\delta=1.1\), the lower endpoint is \(0.01\), and the interval excludes zero at both values. Therefore, \(\delta=1\) is an isolated inclusion point and the first value on the path at which inclusion occurs. The function \((\delta-1)^2\) is positive on both sides of its root, so a sign-change-only search would fail. Checking the endpoint’s minimum or directly evaluating the candidate reveals the tangency.
Interpretation, Comparisons, and Failure Modes
A confidence-interval tipping point is a boundary in the interval-based conclusion under a specified sensitivity path. It does not say that the sensitivity value is likely, that the assumptions at that value are plausible, or that the effect itself has changed direction. It also does not establish a clinical conclusion unless the threshold and the interpretation of the interval support that conclusion. State the confidence level, threshold, direction and start of the path, whether the point is an entry or exit, and what happens at equality.
This differs from a point-estimate tipping point, which tracks when an estimate crosses a threshold. A confidence interval can include a null while its estimate remains on one side; conversely, its estimate can cross a threshold while the interval continues to include values on both sides. These are distinct questions. A p-value may summarize a test conclusion, but a confidence interval additionally shows the estimated range and where the threshold lies. When the interval and test are dual under matching assumptions, the threshold-inclusion rule and test decision agree; they should not be treated as interchangeable if their procedures differ.
Several checks help detect an analysis going wrong:
- Check the initial status. Evaluate both endpoints at the starting value before describing an entry or exit. An interval that already includes the threshold cannot have a later first entry into inclusion on that same path.
- Check equality and direction. Inclusion uses non-strict inequalities. At an endpoint equal to \(c\), the interval includes \(c\); it may exclude \(c\) immediately after an exit boundary.
- Check both endpoints. A root of \(L(\delta)=c\) or \(U(\delta)=c\) is only a candidate. The other endpoint and nearby interval statuses determine what it means.
- Check for multiple roots and tangencies. Nonmonotone endpoints can produce repeated changes, and tangencies can occur without sign changes. Do not claim uniqueness from a single bracket or a coarse grid.
- Check the inference at each sensitivity value. A standard error or interval procedure that ignores uncertainty required by the analysis can move the apparent boundary for the wrong reason.
A final limitation is that the location of the boundary is conditional on the chosen threshold, confidence level, interval construction, and sensitivity path. Changing any of these can move or remove the boundary. The value is a concise summary of one specified analysis, not a universal property of the data. If no boundary occurs over the examined range, report that fact with the range and starting conclusion rather than extrapolating beyond it.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
- A 95% interval is \([1-0.5\delta,\;3-0.5\delta]\) for \(\delta\geq0\). For threshold \(c=0\), find the first inclusion boundary and state the interval conclusion at equality.
- An interval begins at \([2,6]\) for threshold \(c=4\), and its lower endpoint increases continuously. Is it accurate to call a later endpoint contact the “first entry into inclusion”? Explain.
- Why can checking only for a sign change in \(L(\delta)-c\) miss a value at which the interval includes \(c\)? Give a function illustrating the issue.
- At a proposed lower-endpoint boundary, what must be checked besides the equation \(L(\delta)=c\)?
Solutions. (1) Initially \(L(0)=1>0\), so the interval excludes zero. Solve \(1-0.5\delta=0\), giving \(\delta=2\). At equality the interval is \([0,2]\), which includes zero; just above 2, the lower endpoint is negative and the upper endpoint remains positive, so inclusion continues. Thus this is the first entry, not an exit. (2) No. The interval already includes 4 because \(2\leq4\leq6\). A later lower-endpoint contact, if it occurs while the upper endpoint remains at least 4, is relevant to an exit, not a first entry. (3) A root need not change sign: \((\delta-1)^2\) is zero at 1 but positive on both sides. The corresponding endpoint touches the threshold there, which can yield inclusion at that single value. (4) Check that the other endpoint is on the appropriate side of \(c\), and evaluate inclusion immediately before and after the candidate when those path values exist. This establishes whether the contact is an entry, exit, isolated inclusion, or a root that does not change the interval conclusion.