Design Sensitivity and Sample-Size Planning

What Design Sensitivity Measures

A sensitivity analysis asks how strongly hidden bias would have to affect treatment assignment before the study’s conclusion changes. A tipping point describes that change for one particular study and dataset. Design sensitivity asks a different, prospective question: if the study were repeated with more and more matched pairs drawn from a specified treatment-effect distribution, how large a hidden-bias parameter could a chosen test withstand and still detect the effect?

The answer is not a universal property of a study design. It depends on the test, the assumed distribution of treatment effects, and the direction of the alternative. It is also an asymptotic quantity: increasing the sample size can make a test more likely to attain its design sensitivity, but does not by itself change the limiting value.

Definition: For a specified test and a specified distribution of matched-pair effects, the design sensitivity is the limiting value \(\widetilde{\Gamma}\) such that, as the number of independent matched pairs grows, the test has power tending to 1 against the effect when the sensitivity parameter \(\Gamma\) is below \(\widetilde{\Gamma}\), and power tending to 0 when \(\Gamma\) is above it. Behavior exactly at the boundary need not have either limit.

Here \(\Gamma \geq 1\) is the odds-ratio bound in the Rosenbaum model: within a matched pair, the odds of treatment assignment for one unit rather than the other may differ by at most a factor of \(\Gamma\) because of an unmeasured covariate, conditional on the matched information. The worst-case sensitivity analysis uses the assignment probabilities most unfavorable to the hypothesized effect. A larger design sensitivity means greater asymptotic resistance to this particular form of hidden bias.

Assumptions and the Matched-Pair Effect

Consider \(n\) independent matched pairs. Define \(D_i\) as the treated unit’s outcome minus the control unit’s outcome in pair \(i\); positive values favor the prespecified direction of the alternative. For a prospective calculation, the \(D_i\) are modeled as independent draws from a common distribution \(D\). This assumption describes repeated sampling of pairs, not the conditional randomization model used to compute a sensitivity bound for a realized study.

The paired design and the test’s assignment model must be appropriate: pairs are independent, the outcome and treatment are consistently defined, there is no interference between units, and the hidden-bias bound represented by \(\Gamma\) is the sensitivity model being considered. To use the signed-rank derivation below, assume \(P(D=0)=0\), that \(|D|\) has a continuous distribution, and that \(P(D>0)>P(D<0)\). Continuity avoids rank ties in the derivation. In actual data, zero differences and ties require the test’s specified conventions; they should not be silently treated as continuous observations.

These assumptions do not establish that the effect distribution is known. Design sensitivity is conditional on a planning model for that distribution. In particular, its calculation is not an estimate of how much hidden confounding actually exists.

Deriving the Signed-Rank Design Sensitivity

For the one-sided Wilcoxon signed-rank test, let \(A=|D|\), let \(F_A\) be the cumulative distribution function of \(A\), and define \(Q=F_A(A)\). With continuous \(A\), \(Q\) is uniform on \((0,1)\). In a large sample, the rank of a pair’s absolute difference, divided by \(n+1\), is approximately \(Q\). Thus, positive differences with larger magnitudes contribute more to the positive rank sum.

Define \(W^+\) as the sum of the ranks attached to positive differences. Under the assumed effect distribution, its rank-weighted positive share converges to \(q_+ = E[Q I(D>0)]/E[Q]\). Since \(E[Q]=1/2\), the corresponding negative share is \(1-q_+\). Under the worst-case assignment probabilities permitted by the Rosenbaum model, the positive assignment probability for a pair can be as large as \(\Gamma/(1+\Gamma)\). The worst-case expected share of positive ranks is therefore \(\Gamma/(1+\Gamma)\).

For large samples, the observed share exceeds that worst-case null share when \(q_+>\Gamma/(1+\Gamma)\). Solving the equality gives the limiting boundary:

\frac{\Gamma}{1+\Gamma}=q_+ \quad\Longrightarrow\quad \widetilde{\Gamma} =\frac{q_+}{1-q_+} =\frac{E[Q I(D>0)]}{E[Q I(D<0)]}.

The last equality follows because \(D\) is nonzero and \(E[Q]=1/2\), so \(E[Q I(D>0)]+E[Q I(D<0)]=1/2\). This argument compares the large-sample observed rank share with the worst-case null share. As the number of independent pairs grows, sampling fluctuations around these shares become small relative to their separation. If \(\Gamma\) is below the boundary, the observed share is larger; if it is above, the worst-case null share is larger.

Result: For a one-sided signed-rank test under the stated assumptions, design sensitivity is the ratio of the expected rank weight carried by positive effects to that carried by negative effects. It depends on both the signs and the magnitudes of the paired effects.

For comparison, the one-sided sign test gives every nonzero pair the same weight. If \(p_+=P(D>0)\) and \(p_-=P(D<0)\), its design sensitivity is \(p_+/p_-\): under \(\Gamma\), the worst-case null positive-sign probability is \(\Gamma/(1+\Gamma)\), and the effect’s positive-sign probability is \(p_+\). The signed-rank test can therefore have a different design sensitivity from the sign test, even for the same proportion of positive effects.

Worked Example: Design Sensitivity for a Sign Test

Suppose a hypothetical planning model says that a randomly selected matched pair has a positive difference with probability \(0.75\) and a negative difference with probability \(0.25\). Assume differences are independent across pairs and none is zero. What is the sign-test design sensitivity?

For the sign test, \(\widetilde{\Gamma}=p_+/p_-\). Substituting the planning probabilities gives \(\widetilde{\Gamma}=0.75/0.25=3.00\).

Under this model, the one-sided sign test’s power tends to 1 as the number of pairs grows for \(\Gamma<3.00\), and tends to 0 for \(\Gamma>3.00\). This is a statement about the specified effect distribution and test, not evidence that the actual hidden-bias parameter is below 3.00. The calculation also does not use the sizes of the positive and negative differences.

Magnitude Patterns Matter for the Signed-Rank Test

The signed-rank result makes a useful distinction between overall outcome variability and the way effect magnitudes are related to their signs. Multiplying every difference by the same positive constant leaves its ranks and signs unchanged, so it does not change signed-rank design sensitivity. But if positive effects tend to have larger absolute magnitudes than negative effects, more rank weight falls on positive differences. That can increase design sensitivity. If the larger magnitudes tend to be negative, it can decrease it.

Worked Example: Same Sign Proportion, Different Rank Sensitivity

Consider two hypothetical effect distributions. In each, half of the pairs have smaller absolute differences and half have larger absolute differences. Absolute differences in the smaller group lie below all those in the larger group; within each group their values vary continuously. Thus the average value of \(Q=F_A(A)\) is \(0.25\) in the smaller group and \(0.75\) in the larger group. In both distributions, three quarters of all differences are positive.

In distribution A, 60% of the smaller differences and 90% of the larger differences are positive. The expected rank weight on positive differences is \(0.5(0.25)(0.60)+0.5(0.75)(0.90)=0.4125\). The expected rank weight on negative differences is \(0.5(0.25)(0.40)+0.5(0.75)(0.10)=0.08750\). Therefore, \(\widetilde{\Gamma}=0.4125/0.08750=4.714\).

In distribution B, reverse the sign probabilities: 90% of the smaller differences and 60% of the larger differences are positive. The positive rank weight is \(0.5(0.25)(0.90)+0.5(0.75)(0.60)=0.3375\), and the negative rank weight is \(0.5(0.25)(0.10)+0.5(0.75)(0.40)=0.1625\). Thus, \(\widetilde{\Gamma}=0.3375/0.1625=2.077\).

Both distributions have an overall positive-difference probability of \(0.75\), so both give sign-test design sensitivity 3.00. The signed-rank sensitivities differ because distribution A concentrates more positive signs among the higher-ranked differences. This is why a statement such as “greater outcome variability lowers design sensitivity” is not generally correct: the ranking pattern, rather than a common change of measurement scale, determines this signed-rank calculation.

Using Sample Size to Plan a Study

Design sensitivity alone does not specify how many pairs are needed. At a finite sample size, the test may have low power even when the target \(\Gamma\) is below the asymptotic boundary. Sample-size planning therefore needs a target \(\Gamma\), a significance level \(\alpha\), a desired power, and an assumed effect distribution. The sample size controls the precision and power of the test under these assumptions; it does not change the population design sensitivity.

Worked Example: Approximate Sign-Test Sample Size at a Target Gamma

Suppose a hypothetical study plans a one-sided sign test with target \(\Gamma=2\) and \(\alpha=0.05\). The assumed effect distribution has \(p_+=0.75\) and \(p_-=0.25\), and the desired power is \(0.80\). How many independent matched pairs are indicated by a normal-approximation calculation?

At \(\Gamma=2\), the worst-case null probability of a positive sign is \(p_0=\Gamma/(1+\Gamma)=2/3\). Under the assumed effect distribution, the positive-sign probability is \(p_1=0.75\). The expected difference in positive-sign proportions is \(p_1-p_0=0.75-2/3=0.08333\).

A standard one-sided normal-approximation planning formula is \(n \approx [z_{1-\alpha}\sqrt{p_0(1-p_0)}+z_{1-\beta}\sqrt{p_1(1-p_1)}]^2/(p_1-p_0)^2\), where \(1-\beta\) is the target power. Using \(z_{0.95}=1.645\) and \(z_{0.80}=0.842\):

n \approx \frac{ \left[ 1.645\sqrt{(2/3)(1/3)} +0.842\sqrt{(0.75)(0.25)} \right]^2 }{(0.75-2/3)^2} = \frac{(0.7755+0.3646)^2}{0.006944} \approx 187.2.

The approximation indicates about 188 independent matched pairs. The assumed sign probabilities imply a sign-test design sensitivity of \(0.75/0.25=3.00\), so the target \(\Gamma=2\) is below that asymptotic boundary. The planning calculation adds a finite-sample question: whether roughly 188 pairs provide the desired power at that target. Because the sign count is discrete and the normal approximation is only approximate, a final plan should check power using the exact binomial distribution at the worst-case null probability. If some recruited pairs are expected to be unusable, the number enrolled must also exceed the number of analyzable pairs.

Diagnostics, Comparisons, and Limitations

A design-sensitivity calculation is most useful when the effect-distribution assumptions are made visible. For a signed-rank analysis, examine plausible distributions of paired differences, including how often differences are positive and how positive and negative differences are distributed across magnitudes. A calculation based only on an assumed average effect can miss this rank pattern. If the study has many ties or zero differences, revisit the statistic and its rank convention rather than applying the continuous-distribution derivation unchanged.

The sign test is less sensitive to magnitude patterns because it uses signs alone. The signed-rank test uses magnitude ranks and can gain sensitivity when larger ranks tend to have the favorable sign, but its performance depends on that pattern. A mean-based paired test targets a different summary and relies on its own distributional and testing conditions; its design sensitivity is not interchangeable with the signed-rank result. Each test’s design sensitivity is therefore specific to that test and its assumptions.

The method has several limits. The effect distribution used for planning may be wrong; dependence between matched pairs invalidates the independent-pair argument; and finite-sample power can be substantially below its asymptotic behavior. A single design-sensitivity value also does not report uncertainty about the assumed distribution, establish that a particular \(\Gamma\) is plausible, or repair problems in matching or outcome measurement. In practice, compare results across scientifically plausible effect distributions and verify finite-sample power at the planned sample size.

Pitfall: “Doubling the number of pairs doubles design sensitivity.” This confuses asymptotic resistance with finite-sample power. Under a fixed effect distribution, the design sensitivity stays the same as \(n\) changes; more pairs can make the test more likely to reach its limiting performance.
Pitfall: “A larger standard deviation necessarily lowers signed-rank design sensitivity.” A common positive rescaling changes neither ranks nor signs. The relevant issue is how effect signs align with their magnitude ranks, not the measurement scale by itself.
Pitfall: “A design sensitivity of 3 shows that hidden bias is less than 3.” The value is a theoretical boundary under a specified effect distribution and test. It does not measure the actual hidden bias in a particular study.

Key takeaway: Design sensitivity is an asymptotic, test-specific measure of resistance to hidden bias. Effect strength and the sign–magnitude pattern shape it; sample size determines how much finite-sample power is available to approach it.

Check Your Understanding

Try each problem, then check the concise solutions below.

A sign-test planning model has \(P(D>0)=0.80\) and \(P(D<0)=0.20\). Find its design sensitivity.
Does multiplying every nonzero paired difference by 5 change the Wilcoxon signed-rank design sensitivity? Explain.
For a sign test at \(\Gamma=3\), what is the worst-case null probability of a positive sign?
Why can a larger sample improve power without changing design sensitivity?
Name two assumptions or diagnostics that matter when using the signed-rank derivation.

Solutions. (1) \(\widetilde{\Gamma}=0.80/0.20=4.00\). (2) No. Positive rescaling preserves signs and the ordering of absolute differences, so the rank weights and their positive-to-negative ratio are unchanged. (3) \(\Gamma/(1+\Gamma)=3/4=0.750\). (4) More pairs reduce sampling fluctuations, making the test more likely to distinguish the effect’s rank or sign share from the worst-case null share; the limiting shares themselves do not change. (5) Examples include independence of matched pairs, a continuous distribution of absolute differences for the derivation, no zero differences, and checking how positive signs align with magnitude ranks.