Combining Evidence Across Matched Sets

Why Combine Evidence at the Set Level?

A matched study may contain many sets, each contributing evidence about the same sharp null hypothesis. To assess that evidence together, define an overall statistic from the set-level contributions and determine its conditional distribution under the sensitivity model. The key operation is convolution: each set has its own conditional distribution, and the distribution of their sum is built by adding together all compatible set-level values and multiplying their probabilities.

The sensitivity analysis asks how large the upper-tail probability could be if treatment assignment within each matched set were affected by an unmeasured covariate, subject to a specified bound. The resulting worst-case p-value is not obtained by adding the set-specific p-values. It is the largest probability, over allowed assignment mechanisms, that the overall statistic is at least as extreme as observed.

Definition: Let $T_j$ be the contribution from matched set $j$, and let $T=\sum_{j=1}^J T_j$ be the combined statistic. For a one-sided upper-tail test with observed value $t_{\mathrm{obs}}$, the worst-case conditional p-value at sensitivity parameter $\Gamma$ is $$ p_{\mathrm{worst}}(\Gamma)=\sup_{\mathbf u\in\mathcal U(\Gamma)} P_{\mathbf u}(T\ge t_{\mathrm{obs}}\mid\text{matched sets and fixed treatment counts}), $$ where $\mathcal U(\Gamma)$ is the collection of within-set assignment mechanisms permitted by the sensitivity model. The inequality includes ties.

Assumptions and the Overall Distribution

The target here is a test of the sharp null hypothesis that treatment has no effect on any unit’s outcome. Under that null, each unit’s outcome is fixed regardless of its treatment assignment, so the statistic can be evaluated for every allowed assignment. The matched sets and the number treated in each set are treated as fixed. Conditional on these quantities, assignments in different sets are assumed independent. This conditional independence is what permits the set-level distributions to be convolved.

Under Rosenbaum’s $\Gamma$ model, the odds of treatment for two units in the same set can differ by at most a factor of $\Gamma$, where $\Gamma\ge 1$. For a set $j$ with $n_j$ units and $m_j$ treated units, one standard representation assigns a treatment vector $\mathbf a=(a_1,\ldots,a_{n_j})$, with $a_i\in\{0,1\}$ and $\sum_i a_i=m_j$, probability

P_{\mathbf u_j}(\mathbf A_j=\mathbf a) = \frac{\exp\!\left((\log\Gamma)\sum_{i=1}^{n_j}a_i u_{ji}\right)} {\displaystyle\sum_{\mathbf b:\,\sum_i b_i=m_j} \exp\!\left((\log\Gamma)\sum_{i=1}^{n_j}b_i u_{ji}\right)}, \qquad 0\le u_{ji}\le 1.

Here the $u_{ji}$ represent unmeasured covariate values on a scale from 0 to 1. At $\Gamma=1$, assignments with the fixed treated count are equally likely. For $\Gamma>1$, the odds of treatment may vary within a set, but the model limits how much they can vary.

Write $f_j(t)=P(T_j=t)$ for a set’s conditional distribution at a specified allowed assignment mechanism. Independence gives the combined distribution by the recurrence

g_0(0)=1,\qquad g_j(t)=\sum_s g_{j-1}(t-s)f_j(s),\quad j=1,\ldots,J.

The sum is over possible values $s$ of $T_j$; values not in the support have probability zero. Thus $g_J(t)=P(T=t)$. For a fixed choice of the hidden-bias parameters, the upper-tail probability is $\sum_{t\ge t_{\mathrm{obs}}}g_J(t)$. Sensitivity analysis then maximizes that quantity over the allowed parameters in all sets. A set’s maximizing parameters can depend on the overall tail event, so in general one must optimize the combined tail probability, not optimize a separate p-value for each set and combine those p-values.

Matched Pairs: A Useful Special Case

For a matched pair, suppose exactly one member is treated. Let $X_j=1$ when the treated member is the one with the higher fixed outcome score, and $X_j=0$ otherwise. The treated member’s chance of being the higher-scoring member can be as large as

q_\Gamma=\frac{\Gamma}{1+\Gamma}.

For a one-sided test in which larger values of $T=\sum_j w_jX_j$ are more extreme, with fixed nonnegative weights $w_j$, the worst case puts this probability at $q_\Gamma$ for each pair. The pair indicators are conditionally independent, so the distribution is a weighted sum of independent Bernoulli variables. This simple form is especially convenient for hand calculations. For more general set statistics, however, the conditional distribution may have several possible contributions per set, and the full set-level assignment model is needed.

Worked Example: A Worst-Case Tail Probability for Three Pairs

Suppose three matched pairs contribute weights $1$, $2$, and $3$. The observed statistic is $t_{\mathrm{obs}}=5$, and consider $\Gamma=2$. Under the sharp null and the pair sensitivity model, each pair has probability at most

$q_\Gamma=2/(1+2)=2/3$

of contributing its positive weight. The worst-case upper-tail distribution uses probability $2/3$ for each positive contribution. To reach at least 5, the pair with weight 3 and the pair with weight 2 must both contribute: together they give 5, and adding the weight-1 pair still gives a value at least 5. The weight-1 pair’s assignment is unrestricted by this event. Therefore

$P(T\ge5)=(2/3)(2/3)=4/9=0.4444.$

Thus the worst-case one-sided p-value at $\Gamma=2$ is 0.4444. It is the largest conditional probability of obtaining a statistic at least 5 under the sharp null and the specified sensitivity bound. It is not the probability that the null is true. At $\Gamma=1$, the corresponding probability is $(1/2)(1/2)=0.25$, as expected under equal-probability pair assignments.

Convolving Variable-Ratio Sets

In a variable-ratio matched set, the set-level statistic may take more than two values. First calculate the probability of each possible contribution by summing the probabilities of assignments that produce it. Then convolve this distribution with the distributions from the other sets. If there are only a few assignments, direct enumeration is transparent; for larger studies, a dynamic-programming implementation of the recurrence is more efficient.

Worked Example: One Three-Unit Set Combined With a Pair

Consider a set of three units with one treated. Their fixed outcome scores are 0, 1, and 3, and the set contributes the score of the treated unit. At $\Gamma=2$, take sensitivity values $u=(0,0,1)$, assigning the largest sensitivity value to the unit with score 3. The assignment weights are $1,1,2$, so the probabilities for contributions 0, 1, and 3 are $1/4,1/4,1/2$, respectively.

Now add an independent matched pair contributing 2 if its higher-scoring member is treated and 0 otherwise. Its worst-case probabilities at $\Gamma=2$ are $2/3$ for contribution 2 and $1/3$ for contribution 0. For an overall threshold of 3, the combined statistic reaches the threshold either when the three-unit set contributes 3, or when it contributes 1 and the pair contributes 2. Hence

$P(T\ge3)=(1/2)(1)+(1/4)(2/3)=1/2+1/6=2/3.$

For this example, $2/3$ is the largest probability over the permitted sensitivity values for the three-unit set. To check this, write its assignment weights as $w_0,w_1,w_3\in[1,2]$. The tail probability is $(w_3+(2/3)w_1)/(w_0+w_1+w_3)$. Its maximum over these bounds is $2/3$, attained, for example, at $(w_0,w_1,w_3)=(1,1,2)$ or $(1,2,2)$. This calculation also illustrates why the relevant quantity is the probability of the combined tail event: it is not a product or sum of two separately calculated p-values.

A General Enumeration and Optimization Procedure

For each set, list the permitted assignments and calculate both the assignment probability and the statistic contribution under each one. Group assignments with the same contribution to get the set-level distribution. For a specified collection of sensitivity values, use the convolution recurrence to obtain the overall tail probability. Finally, maximize that probability across the allowed sensitivity values. For a small number of units, enumeration can verify the calculation; for larger problems, dynamic programming avoids listing every cross-set combination.

Fix the test and tail.
Specify the sharp null, the set contributions, the direction of the test, and the observed threshold $t_{\mathrm{obs}}$.

Find each set’s distribution.
For a given allowed sensitivity assignment, enumerate its possible treatment allocations, calculate their probabilities, and group allocations by their statistic contribution.

Combine the distributions.
Convolve the set-level distributions, or use the recurrence for $g_j(t)$, to obtain $P(T\ge t_{\mathrm{obs}})$.

Maximize over hidden bias.
Search over the allowed sensitivity values and retain the largest combined tail probability. Verify that the selected values satisfy the same $\Gamma$ bound in every set.

In a discrete distribution, the tail sum must include every value equal to the observed statistic. Omitting ties makes the reported p-value too small. If a lower-tail test is intended, use $P(T\le t_{\mathrm{obs}})$ and maximize that lower-tail probability under the same model. For a two-sided test, define the two-sided extremeness rule explicitly; simply doubling a worst-case one-sided p-value is not generally guaranteed to give the exact worst-case two-sided p-value.

Worked Example: Convolution With Unequal Contributions

Suppose two independent sets have contributions 0 or 1 and 0 or 2, respectively. Under a specified allowed assignment mechanism, let the probabilities of the positive contributions be $0.6$ and $0.7$. If the observed combined statistic is 3, both contributions must be positive. Convolution gives

$P(T=3)=P(T_1=1)P(T_2=2)=0.6(0.7)=0.42.$

This is the upper-tail probability $P(T\ge3)$ for this mechanism. For a sensitivity analysis, 0.42 is not automatically the worst-case p-value: the calculation must be repeated, or optimized, over all assignment mechanisms allowed by $\Gamma$. The example makes the independence requirement visible: multiplying these probabilities is justified because the sets’ assignments are conditionally independent, not merely because the data were divided into sets.

Diagnostics, Comparisons, and Limitations

Several checks help catch errors before interpreting a sensitivity bound. For every set, verify that assignment probabilities are nonnegative and sum to 1. After each convolution, check that the combined probabilities sum to 1, apart from minor rounding error, and that the support contains only possible sums of set contributions. At $\Gamma=1$, confirm that the assignment probabilities reduce to equal probabilities among allocations with the fixed treated count. As $\Gamma$ increases, the worst-case upper-tail probability cannot decrease, because a larger $\Gamma$ permits all mechanisms allowed by a smaller one as well.

A direct enumeration is a useful diagnostic for small examples, even when software is used for the full analysis. If a reported worst-case tail probability is below the value calculated at one permitted choice of sensitivity parameters, the optimization is wrong: a maximum cannot be smaller than one of its feasible values. For a matched-pair upper-tail statistic with nonnegative weights, using $q_\Gamma$ for each pair provides a simple check. That shortcut does not automatically apply to every set-level statistic.

The main alternative to set-level convolution is to use a large-sample approximation for the combined statistic. Such an approximation can be useful when there are many contributions, but it may be inaccurate with few sets, discrete contributions, or a tail probability driven by a small number of large scores. Convolution preserves the conditional discrete distribution, at the cost of more computation. Combining separately reported p-values is a different procedure and generally does not calculate the probability that this study’s combined statistic reaches its observed value.

The result is conditional on the matched sets, treatment counts, sharp null, and sensitivity model. It does not show that an unmeasured covariate actually has a particular value, nor does a non-small worst-case p-value establish that hidden bias explains the observed association. The bound is also only as appropriate as the chosen statistic, tail definition, and $\Gamma$ range. In particular, a test of the sharp null is not by itself an estimate or confidence interval for a nonzero treatment effect.

Do not combine p-values as if they were set contributions. The overall tail probability comes from the distribution of $T=\sum_jT_j$.
Do not optimize one tail when testing another. The worst-case mechanism depends on the specified tail event.
Do not ignore the assignment structure. Fixed treated counts and the conditional assignment probabilities determine each set’s distribution.
Do not assume every set has a Bernoulli distribution. Variable-ratio sets can have several possible statistic contributions.

Key takeaway: Under conditional independence across matched sets, combine set-level conditional distributions by convolution, then maximize the probability of the prespecified overall tail event over all assignment mechanisms allowed by $\Gamma$.

Check Your Understanding

Answer each question before reading the solutions.

Why can the set-level distributions be convolved in this analysis?
For a matched pair with $\Gamma=3$, what is the largest permitted probability that the higher-scoring member is treated?
Two independent sets contribute 0 or 1 with probability 0.4 of 1, and 0 or 2 with probability 0.5 of 2. What is the probability that their sum is at least 3 under this specified mechanism?
Why is the worst-case combined p-value not generally the sum of the set-specific p-values?
What should happen to the worst-case upper-tail p-value as $\Gamma$ increases?

Solutions. (1) Conditional on the matched sets and fixed treatment counts, assignment is assumed independent across sets; independence makes the distribution of the sum a convolution. (2) $q_\Gamma=3/(1+3)=0.75$. (3) The sum reaches 3 only when both sets contribute positively, so the probability is $0.4(0.5)=0.20$. (4) Set-specific p-values are not contributions to the overall statistic’s distribution. The combined p-value is the maximum, over permitted assignment mechanisms, of the probability that the combined statistic reaches its observed threshold. (5) It cannot decrease: increasing $\Gamma$ enlarges the set of allowed assignment mechanisms, so the maximum is taken over a superset of the previous mechanisms.