What a Sensitivity Bound Answers
A matched observational study can make treated and control units look similar on measured covariates, yet leave uncertainty about unmeasured differences. Rosenbaum sensitivity analysis asks how conclusions from a test could change if treatment assignment within matched sets depended on an unmeasured factor. Its central output is a bound on a test’s tail probability under a specified assignment model. That is not automatically an estimate, or an interval, for an average causal effect.
The distinction matters because a hypothesis test and an effect estimate answer different questions. A sensitivity analysis may show that a test of a particular sharp null remains unlikely across a range of hidden-bias strengths. That finding concerns the compatibility of the data with that null and assignment model. To identify an average effect, one also needs assumptions or design information that connect the observed outcomes to the unobserved potential outcomes.
A null is sharp when it specifies enough to fill in every missing potential outcome under that null. For the sharp null of no effect, each unit’s observed outcome is its outcome under either assignment. Given a possible treatment assignment, the test statistic can therefore be calculated under the null. The randomization distribution is formed by considering assignments allowed by the design or assignment model.
In a Rosenbaum sensitivity model, treatment odds for units in a matched set may differ because of an unmeasured covariate, but their relative odds are restricted. The sensitivity parameter \(\Gamma \geq 1\) bounds the relevant odds ratio; \(\Gamma=1\) corresponds to equal treatment odds within the set. For a test statistic and null hypothesis, the analysis finds a worst-case tail probability over assignment probabilities consistent with that restriction. The resulting upper bound is a conservative p-value bound under the model, not a probability that the null is true.
Why a Test Bound Does Not Reveal an Average Effect
The assignment restriction controls how treatment may depend on hidden factors. It does not specify how those factors affect outcomes, determine the missing potential outcomes, or impose a common treatment effect across units. Thus it does not generally identify the average of the unit-level effects. In particular, the sharp null test can be carried out because that null supplies the missing outcomes for the purpose of testing; the sensitivity model does not supply them under alternatives.
This gap can be seen directly. In a matched pair, one unit is treated and the other is a control. The observed treated outcome is \(Y_i(1)\) for the treated unit, while the observed control outcome is \(Y_j(0)\) for the control unit. Their difference combines outcomes from two different people. It is not, without further assumptions, the treated person’s effect \(Y_i(1)-Y_i(0)\), nor the pair’s average effect.
Worked Example: The Same Matched Data, Different Effects
Suppose four matched pairs have the following observed outcomes. In each pair, A is treated and B is a control.
| Pair | A: observed treated outcome | B: observed control outcome | Observed difference |
|---|---|---|---|
| 1 | 8 | 4 | 4 |
| 2 | 7 | 5 | 2 |
| 3 | 6 | 5 | 1 |
| 4 | 5 | 6 | -1 |
The mean observed treated-minus-control difference is
But this contrast does not reveal the average treatment effect among the four treated A units, because their untreated outcomes are unobserved. One possible completion of those missing outcomes is \(Y_A(0)=(7,7,6,5)\). The corresponding effects on A are \((1,0,0,0)\), with average \(0.25\). Another possible completion is \(Y_A(0)=(3,3,2,1)\). The effects are then \((5,4,4,4)\), with average \(4.25\).
Both completions leave every observed outcome, and hence the observed mean difference of \(1.5\), unchanged. The assignment odds restriction alone does not choose between these unobserved outcomes. Consequently, a sensitivity bound for a sharp-null test cannot distinguish these average effects merely by being calculated at a particular \(\Gamma\).
This is an identification issue, not a problem solved by collecting a more elaborate p-value. Even if a test rejects the sharp null, that does not identify the size of an average effect: the sharp null rules out every nonzero unit effect, while rejection can be compatible with many patterns of positive and negative effects. Conversely, failure to reject does not establish that the average effect is zero.
A Bound on a Sharp-Null Test
For a fixed sharp null, the outcomes that would have been observed under alternative assignments are determined by the null. A sensitivity analysis then asks how extreme the chosen statistic could be under assignment probabilities allowed by \(\Gamma\). For example, a worst-case upper-tail probability is the largest probability of obtaining a statistic at least as extreme as the observed one, among those allowed assignment distributions. If this upper bound is small, the data are difficult to reconcile with that sharp null across the specified range of hidden bias.
A simple sign-test calculation illustrates the meaning of the bound. Assume there are four matched pairs, no ties, and treatment assignments are conditionally independent across pairs. Under the sharp null, a positive or negative within-pair sign reflects assignment orientation, not a nonzero causal effect. With pairwise odds ratios bounded by \(\Gamma=2\), the probability of the orientation that makes a sign positive can be as large as \(q=2/(1+2)=2/3\). If three of the four observed signs are positive, the worst-case upper-tail probability for the number of positive signs is:
Worked Example: A Worst-Case Sign-Test Probability
For \(X\), the number of positive signs, the upper-tail probability is maximized when each pair has positive-sign probability \(q=2/3\). Thus \(X\) has a binomial distribution with four trials and success probability \(2/3\) for this worst-case calculation.
This is an upper bound on the one-sided tail probability for this sign statistic under the stated sharp null, pairwise sensitivity restriction, and independent-pair assignment assumption. It is not an estimate of an effect of \(0.593\), nor does it say there is a 59.3% probability that the null is true. The example also shows that a sensitivity bound can be uninformative for rejecting the null even when most observed signs point in the same direction.
The calculation depends on its stated setup. With dependence across pairs, the binomial distribution need not describe the assignment distribution; with tied outcomes, the sign statistic needs a prespecified treatment of ties. More generally, the test statistic, tail direction, matched-set assignment model, and assumptions used to obtain the worst-case bound must all be reported. The value of \(\Gamma\) is a sensitivity parameter, not an empirically verified measure of the actual hidden bias.
When Matched Comparisons Can Estimate an Effect
Estimating an average effect requires assumptions that address the missing potential outcomes. For an average treatment effect among treated units, a typical conditional identification argument assumes consistency (the observed outcome equals the potential outcome under the treatment received), no interference between units, and conditional exchangeability of the untreated potential outcome among treated and control units given measured covariates \(X\). It also requires positivity: at covariate values represented among treated units, there must be a positive chance of observing controls. Matching is a design strategy for making these assumptions more plausible with respect to measured \(X\); it does not prove them, particularly for unmeasured confounding.
Under these assumptions, the average effect among treated units can be expressed as the treated mean outcome minus the treated-covariate-distribution average of the conditional control mean:
In a matched sample, the second term may be estimated using matched controls, with appropriate weighting if matching or the target population requires it. This is an effect-estimation strategy that relies on exchangeability and overlap, not a consequence of the Rosenbaum p-value bound. If only a bound on treatment odds is available and the actual assignment probabilities are unknown, it does not by itself supply the outcome model or exchangeability conditions needed for point identification.
Worked Example: Estimating an ATT Under Additional Assumptions
Suppose two treated units and two matched controls have identical measured covariates within each pair. The observed treated outcomes are 8 and 6; the observed control outcomes are 5 and 3. The matched estimate of the average treatment effect among the treated is the mean treated outcome minus the mean matched-control outcome:
The estimate is 3 outcome units. Interpreting it as an estimate of a causal ATT requires, among other conditions, that the matched controls represent the treated units’ untreated outcomes after conditioning on the matched covariates, along with consistency, no interference, and adequate overlap. Exact agreement on measured covariates alone does not ensure that unmeasured factors are balanced. A Rosenbaum sensitivity analysis can assess how a specified hidden-assignment bias affects a sharp-null test, but it does not verify these identification assumptions or turn the estimate into a point-identified effect.
Constant-Effect Inversion and Its Limit
A related procedure tests a sharp null of a specified constant additive effect, \(Y_i(1)=Y_i(0)+\tau_0\) for every unit, and inverts tests across candidate values of \(\tau_0\). This can produce a sensitivity interval for a constant-effect parameter under the corresponding model. The constant-effect hypothesis is sharp because it imputes missing outcomes once \(\tau_0\) is specified.
That interval should not automatically be called an interval for the average treatment effect when effects may vary across units. Inverting tests of constant effects addresses the values of a common effect that are compatible with the test procedure; a heterogeneous average effect is a different estimand. Its identification or interval estimation requires assumptions and methods suited to that estimand. The distinction is especially important when effects can differ in magnitude or direction.
| Analysis | What it addresses | What it does not establish on its own |
|---|---|---|
| Rosenbaum bound for a sharp-null test | Worst-case tail probability under a hidden-bias assignment restriction | An average effect, the null’s probability, or the actual amount of hidden bias |
| Inversion for a constant additive effect | Values of a common effect not rejected by the specified sensitivity tests | An interval for a heterogeneous average effect without additional justification |
| Matched effect estimator | An estimate of a stated causal estimand under identification assumptions | Protection from unmeasured confounding simply because matching was used |
Diagnostics, Pitfalls, and Limitations
A useful report separates the estimand, the test, and the sensitivity model. State which null is tested, what statistic is used, what \(\Gamma\) represents, and whether the reported quantity is an upper-bound p-value or an effect estimate. Check that the matched sets and assignment assumptions used in the calculation correspond to the design. Also examine whether the effect estimate depends heavily on a few matches, whether there is adequate overlap, and whether the measured covariate balance supports the proposed comparison. These checks can reveal design or modeling problems, but they cannot establish that unmeasured confounding is absent.
A common mistake is to reason, “The worst-case p-value is small, so the average treatment effect is identified and positive.” The correction is that the bound concerns a test of a specified null under a specified assignment model. It does not uniquely determine missing potential outcomes or an average effect. A small bound can support evidence against a sharp null over the considered range of \(\Gamma\), but effect magnitude and target-population interpretation require a separate estimation argument.
Sensitivity analysis is valuable precisely because its assumptions are limited and explicit: it asks how a test conclusion changes under bounded hidden assignment bias. Its limitations are equally important. A chosen \(\Gamma\) does not guarantee that the true assignment mechanism satisfies the model; the result concerns the chosen statistic and null; and sensitivity to assignment odds alone may not capture every source of misspecification. Treat the result as a qualified robustness statement, not as a substitute for identifying assumptions.
Check Your Understanding
Answer each question, then compare with the solutions.
- What makes a null hypothesis sharp, and why does the sharp null of no effect permit a randomization test?
- A matched study has a small worst-case p-value for the sharp null at a stated \(\Gamma\). What does that establish, and what does it not establish about the average effect?
- Why can an observed matched-pair mean difference differ from the average effect among treated units?
- What additional assumptions support interpreting a matched comparison as an estimate of the ATT?
- Why might an interval obtained by inverting constant-effect tests not be an interval for a heterogeneous average effect?
Solutions. (1) A sharp null specifies each unit’s treatment effect sufficiently to impute the missing potential outcome under every assignment; under no effect, both potential outcomes equal the observed outcome. This makes the statistic calculable across assignments. (2) It gives evidence against that sharp null under the specified assignment model and hidden-bias bound. It neither identifies the average effect nor gives the probability that the null is true. (3) The contrast subtracts one person’s control outcome from another person’s treated outcome, rather than observing both potential outcomes for the treated person. (4) Consistency, no interference, conditional exchangeability for the relevant untreated potential outcomes given measured covariates, and positivity are among the needed assumptions. (5) Constant-effect inversion concerns a common effect; heterogeneous effects need not equal that common value, so the resulting interval does not automatically cover their average.