Stratified Tests for Matched Binary Outcomes

Why Discordant Pairs Carry the Test

For a binary outcome, a matched pair can be concordant (both units have outcome 0 or both have outcome 1) or discordant (one has outcome 1 and the other has outcome 0). Under the sharp null of no treatment effect for any unit, each unit’s outcome would stay the same if its treatment assignment changed. The two units in a pair may nevertheless have different outcomes from one another. If their outcomes are concordant, switching which unit receives treatment does not change the pair’s treated outcome. If they are discordant, that assignment determines whether the treated unit has outcome 1. Thus, conditional on the outcomes in the pairs, the test is driven by the discordant pairs.

The test below is for matched pairs with one treated and one control unit in each pair. It tests a directional sharp null against the alternative that treatment increases the probability of outcome 1. This may be an adverse event or a beneficial outcome; the direction must be chosen to match the scientific question.

Concordant pairs are not counted in \(m\). They contribute zero to this conditional test statistic: given that both outcomes in a pair are equal under the null, either assignment produces the same treated outcome. The test therefore asks whether the observed allocation of treatment across discordant pairs is unusually favorable to an increase in outcome 1.

Assignment Bounds and the Worst-Case Test

The sensitivity parameter \(\Gamma\) bounds how different the treatment odds may be between the two members of a matched pair because of an unmeasured covariate. At \(\Gamma=1\), the odds are equal within every pair. At \(\Gamma>1\), treatment assignment may favor one member. Under the sharp null, the member with outcome 1 could be the one favored for treatment, making an increase in treated outcomes more likely.

The bound on that probability follows by considering the assignment most favorable to the alternative. If the outcome-1 unit has treatment odds \(\Gamma\) times those of the outcome-0 unit, its probability of receiving treatment is \(\Gamma/(\Gamma+1)\). No assignment allowed by the sensitivity model can give a higher probability. At \(\Gamma=1\), this is \(1/2\); as \(\Gamma\) increases, the bound rises.

The right side is a valid upper bound on the one-sided p-value for the observed result. The actual assignment probabilities may differ across pairs, so \(X\) need not have a binomial distribution. It is a sum of independent Bernoulli variables with probabilities no greater than \(q_\Gamma\). To justify the bound, take independent \(U_i\sim\operatorname{Uniform}(0,1)\) and represent each discordant-pair indicator as \(X_i=1\) when \(U_i\le p_i\), where \(p_i\le q_\Gamma\). The indicators \(1\{U_i\le q_\Gamma\}\) then dominate \(X_i\) pair by pair. Their sum has a \(\operatorname{Binomial}(m,q_\Gamma)\) distribution, so its upper-tail probability is at least as large as the actual upper-tail probability. This gives a conservative worst-case test.

For the directional alternative of an increase, use the binomial upper tail. A small bound means that even the assignment most favorable to hidden bias would rarely produce at least \(x\) pairs favoring an increase under the sharp null. A large bound means the data do not rule out that null at the specified \(\Gamma\). The bound is not the probability that the null is true, nor the probability that a confounder exists.

For a two-sided analysis, specify the two-sided convention before examining results. One conservative option is to calculate the worst-case upper-tail bound for an increase and the corresponding worst-case lower-tail bound for a decrease, then report the smaller one-sided bound doubled and capped at 1. The examples here use a one-sided alternative so that the tail and its interpretation remain explicit.

Calculating the Stratified Statistic

Each matched pair is a stratum. Its contribution to \(X\) is 1 if it is discordant and the treated member has outcome 1, 0 if it is discordant in the opposite direction, and zero to the conditional test if it is concordant. Summing the discordant contributions gives \(x\), while the total number of discordant pairs gives \(m\). Under no hidden bias, \(X\) has the familiar conditional matched-pair distribution \(\operatorname{Binomial}(m,1/2)\). Sensitivity analysis replaces \(1/2\) with the larger worst-case probability \(q_\Gamma\).

Diagnostics, Comparisons, and Limitations

Before calculating a bound, verify that the data are organized as matched pairs with one treated unit and one control unit per pair, that the outcome is coded consistently, and that the direction of the alternative is stated. Count the two discordant directions separately and check that their sum equals \(m\). Concordant pairs should not be added to the binomial denominator. If \(m=0\), there is no discordant-pair information for this conditional test; the statistic cannot provide evidence in either direction.

The exact binomial tail is preferable to a normal approximation when the number of discordant pairs is small or the observed count is near 0 or \(m\). At \(\Gamma=1\), the test is the exact conditional matched-pair test often called McNemar’s exact test. A large-sample score or normal approximation can be useful for a rough check, but it may be inaccurate in small samples and does not replace the exact worst-case calculation. Unlike a test based on independent treatment and control proportions, this analysis conditions on the matched-pair structure and uses the within-pair discordances.

The sensitivity bound is conservative: it permits each discordant pair’s assignment probability to take the value most unfavorable to the null test, whether or not a single realistic unmeasured covariate would produce that pattern across all pairs. Its interpretation depends on the sharp null, the matched-pair assignment model, and the independence condition across pairs. It is not a bound on every causal estimand, such as an average treatment effect, and it does not correct measured confounding or poor matching. The analysis also cannot reveal whether an unmeasured confounder of a given strength is actually present.