From a Sensitivity Test to an Effect Interval
A sensitivity test asks whether a particular treatment-effect value remains compatible with the observed matched data after allowing for hidden bias of a specified strength. Inverting that test means repeating the test across candidate effect values and retaining the values that are not rejected. The result is a sensitivity-adjusted confidence set: its interpretation depends on both the effect model and the chosen hidden-bias bound.
This is not the same as reporting a point estimate and attaching an ordinary standard error. Each candidate effect defines a different null hypothesis, and the sensitivity analysis changes the assignment probabilities allowed under that null. The introductory material on Rosenbaum’s Gamma model gives the hidden-bias parameter’s interpretation; here the focus is how candidate-effect tests combine to form an interval.
The constant-effect assumption is substantive: it says every unit has the same additive effect on the outcome scale. Under \(H_0(\tau_0)\), subtracting \(\tau_0\) from a treated-minus-control pair difference removes the hypothesized effect. The sign of the adjusted difference \(D_i-\tau_0\) is then the pair-level evidence relevant to testing that sharp null. A sign test uses only whether the adjusted difference is positive or negative; exact ties contribute no sign evidence.
The sharp null makes the randomization calculation possible: under the null, the outcomes that would be observed under either assignment are determined by the observed outcomes and the hypothesized effect. The sensitivity model then bounds the chance that either member of a matched pair receives treatment. For a pair, the odds of treatment for one member relative to the other are bounded by \(\Gamma\), where \(\Gamma\geq1\). Consequently, the treatment probability for either member lies between \(1/(1+\Gamma)\) and \(\Gamma/(1+\Gamma)\). At \(\Gamma=1\), treatment is equally likely within each pair; larger \(\Gamma\) permits more imbalance from hidden bias.
Directional Tests and Inversion
For a candidate \(\tau_0\), let \(m(\tau_0)\) be the number of non-tied adjusted pair differences, and let \(K(\tau_0)\) be the number that are positive. Under the sharp null, the sign probability for a pair can vary with its hidden assignment probability. To obtain a conservative upper-tail test, use the largest allowed positive-sign probability \(p_{\max}=\Gamma/(1+\Gamma)\). For a lower-tail test, use the smallest allowed positive-sign probability \(p_{\min}=1/(1+\Gamma)\). The resulting worst-case one-sided p-values for the sign statistic are
Here \(p_+\) tests for evidence that the effect is greater than \(\tau_0\), and \(p_-\) tests for evidence that it is less than \(\tau_0\). These are worst-case p-values: they use the assignment probabilities within the sensitivity model that make the corresponding observed tail least surprising under the null. For a two-sided level-\(\alpha\) confidence set, a common equal-tailed procedure rejects a candidate if either directional p-value is at most \(\alpha/2\). It retains the candidate only if both are greater than \(\alpha/2\). The same tail allocation must be used consistently throughout the inversion.
The reason inversion yields a confidence set is direct. If the true constant effect is \(\tau\), then \(H_0(\tau)\) is true. A level-\(\alpha\) two-sided test rejects that true value with probability at most \(\alpha\) under the assumed sensitivity model. Therefore the set formed by retaining nonrejected values contains the true value with probability at least \(1-\alpha\), subject to the model and test’s validity. Exact tests are often conservative, so actual coverage can exceed the nominal level.
Worked Example: Checking a Candidate Effect at a Tie
Worked Example: Checking a Candidate Effect at a Tie
Suppose ten hypothetical matched pairs have continuous outcomes measured in days, and a sign-test inversion is conducted at \(\Gamma=1\) with \(\alpha=0.05\). At candidate effect \(\tau_0=8\) days, one pair has \(D_i=8\), so its adjusted difference is zero and it is excluded. Suppose the other nine adjusted differences include \(K=2\) positive values and seven negative values. Calculate the lower-direction p-value and decide whether this candidate is rejected.
At \(\Gamma=1\), both allowed sign probabilities equal \(1/2\). Thus \(m=9\), \(K=2\), and the lower-tail p-value is
The lower-tail p-value exceeds \(\alpha/2=0.025\), so this test does not reject the candidate effect of 8 days. In particular, the tied pair is excluded rather than counted as either positive or negative. Treating it as a negative sign would change the test statistic and would not be the stated sign-test procedure.
Worked Example: A Candidate Effect Excluded at the Lower Endpoint
At another candidate, \(\tau_0=2\) days, suppose one pair is tied and is excluded, leaving \(m=9\) non-tied pairs, of which \(K=8\) have positive adjusted differences. Again let \(\Gamma=1\) and \(\alpha=0.05\). Test the candidate in the direction of effects greater than 2 days.
The upper-tail p-value is
Since \(0.01953<0.025\), the candidate value 2 days is rejected by the equal-tailed two-sided procedure. In a hypothetical inversion whose remaining candidate tests retain values above 2 through 8 days and reject values above 8 days, the resulting set is \((2,8]\) days: 2 is excluded and 8 is included. The calculations at the boundaries show why endpoint inclusion cannot be inferred from a rounded interval alone; it follows from the candidate-specific tests, with ties handled at each candidate.
Why the Binomial Tails Are Conservative
For a fixed candidate effect, consider the event that the positive-sign count is at least the observed \(K\). Increasing the positive-sign probability makes this upper-tail event more likely. Since every pair’s probability is at most \(p_{\max}\), using \(p_{\max}\) for each of the \(m\) independent matched-pair sign contributions gives an upper bound on the probability of this event. The analogous argument applies to the lower tail: decreasing the positive-sign probability makes an unusually small count more likely, so \(p_{\min}\) supplies the conservative bound.
This argument relies on independent assignment contributions across matched pairs under the conditional randomization model and on the stated odds bound applying to every pair. It is a justification for this matched-pair sign-test bound, not a universal formula for every sensitivity statistic. For example, a test based on ranks or a stratified binary-outcome statistic requires its own test statistic and worst-case tail calculation. Inverting those tests still follows the same broad logic—test each candidate sharp null and retain nonrejected values—but their p-values should not be replaced by the binomial sign-test formulas above.
Worked Example: Increasing the Hidden-Bias Bound
For a hypothetical candidate effect, suppose six of six non-tied pair differences are positive, so \(m=6\) and \(K=6\). At \(\Gamma=2\), the conservative upper-tail probability uses \(p_{\max}=2/3\). Calculate the upper-tail p-value and compare it with the 0.025 one-sided threshold.
Because all six signs are positive, the event is \(K_{\mathrm{null}}\geq6\), which requires six positive signs:
This candidate is not rejected by the upper-tail test at 0.025. Under \(\Gamma=1\), the corresponding probability would be \((1/2)^6=0.01563\), which is below 0.025. Allowing more hidden bias makes the relevant extreme sign pattern less surprising under the null, so evidence against the candidate weakens. This illustrates why sensitivity-adjusted confidence sets generally become at least as accommodating as \(\Gamma\) increases, although exact endpoint behavior depends on the statistic and discreteness.
Reading the Interval and Diagnosing Problems
A sensitivity-adjusted interval is conditional on three choices: the effect scale and model, the test statistic, and \(\Gamma\). An interval in days under a constant additive effect is not automatically an interval for an average treatment effect, a risk difference, or a heterogeneous collection of individual effects. If effects vary across units, the sharp constant-effect null may be scientifically implausible; the interval then describes values not rejected under that working model rather than resolving effect heterogeneity.
The interval can be discrete, asymmetric, or have endpoints that are included on one side and excluded on the other. Sign counts change only when a candidate crosses an observed pair difference, while tied differences are omitted at the candidate value itself. Therefore the p-value can jump at observed differences. A reported closed interval may conceal these details; when endpoint inclusion matters, evaluate the test at the endpoint rather than relying on a plot or rounding.
A useful implementation audit is to recompute \(D_i-\tau_0\) for every candidate, explicitly count positive, negative, and tied values, and then calculate both directional p-values using the same \(\Gamma\) and tail allocation. Also check that the procedure’s assumptions match the data: independent matched-pair assignment contributions, a valid matched-pair sensitivity bound, and outcomes that support the additive effect model. A candidate should not be rejected merely because a conventional estimate lies outside an interval computed under a different model or test.
| Approach | What it answers | Important distinction |
|---|---|---|
| Inverted sensitivity test | Which candidate effects are not rejected under a specified \(\Gamma\)? | Produces a set tied to the sharp effect model and test statistic. |
| Ordinary matched-pair confidence interval | What effects are compatible under its sampling or randomization assumptions? | Does not automatically protect against hidden bias beyond those assumptions. |
| Sensitivity analysis at a single null | Does one specified effect remain rejected as \(\Gamma\) changes? | Does not by itself provide an interval over effect values. |
These approaches are complementary, not interchangeable. Inversion gives an interpretable range of candidate effects for a fixed sensitivity model, but it can be conservative and can yield irregular sets because tests are discrete. It does not identify the true hidden confounder, prove that the chosen \(\Gamma\) is adequate, or guarantee robustness to misspecification of the constant-effect model. Report the effect scale, \(\Gamma\), test, confidence level, and endpoint convention together.
Common Mistakes
- Counting a tie as a negative sign. In the sign-test procedure defined here, a pair with \(D_i-\tau_0=0\) is removed from \(m\) and from \(K\). Changing the tie rule changes the test.
- Inverting the wrong null. The candidate-specific null is the sharp constant-effect statement \(H_0(\tau_0)\), not simply a null about an average effect unless a different procedure has been established.
- Using an ordinary p-value. The sensitivity-adjusted calculation uses probabilities allowed by the chosen \(\Gamma\), not automatically one-half for every pair.
- Calling every nonrejected value “true.” Nonrejection means the specified test does not rule out that candidate at the chosen level and sensitivity bound; it does not establish the candidate as the effect.
- Assuming both endpoints are included. Exact p-values, ties, and discrete jumps determine whether each boundary is retained. Check each endpoint directly.
Check Your Understanding
Answer each question, then compare with the solutions below.
- At a candidate effect, \(m=8\), \(K=7\), \(\Gamma=1\), and \(\alpha=0.05\). What is the upper-tail p-value, and does that directional test reject?
- At \(\Gamma=3\), what are \(p_{\min}\) and \(p_{\max}\) for the sign-test bounds?
- Why is a pair with \(D_i=\tau_0\) excluded from the sign count in the procedure described here?
- What does it mean for a candidate effect to be retained in an inverted sensitivity confidence set?
Solutions. (1) \(P\{\operatorname{Bin}(8,0.5)\geq7\}=[\binom{8}{7}+\binom{8}{8}]/2^8=(8+1)/256=0.03516\). Since this exceeds 0.025, the upper-tail test does not reject. (2) \(p_{\min}=1/(1+3)=0.25\) and \(p_{\max}=3/(1+3)=0.75\). (3) Its adjusted difference is zero, so it supplies neither a positive nor a negative sign and contributes no evidence to this sign statistic. (4) The two prespecified directional tests both have p-values above \(\alpha/2\); this is failure to reject the candidate under the stated model and sensitivity bound, not proof that it is the true effect.