Any-Rejection and All-Rejections Power

Why Two Power Criteria Are Needed

A multiple-contrast procedure may be judged by two different questions. If a study targets several effects, is it enough to detect at least one of them, or must it detect every one? These questions define disjunctive power and conjunctive power. They are different probabilities, even when they concern the same contrast tests.

For example, a researcher may consider three prespecified treatment comparisons important. The probability of detecting at least one comparison describes the chance of finding some evidence among those targets. The probability of detecting all three describes the chance of a complete set of discoveries. Both depend not only on each contrast’s marginal power but also on the dependence among the test statistics.

Definition: Let $A$ be a nonempty set of indices for the effects of interest, and let $R_j$ denote the event that the test for contrast $j$ rejects its null hypothesis. The disjunctive power for $A$ is $$ P\left(\bigcup_{j\in A} R_j\right), $$ the probability of at least one target effect being detected. The conjunctive power for $A$ is $$ P\left(\bigcap_{j\in A} R_j\right), $$ the probability that every target effect is detected.

The set $A$ specifies which detections count toward the criterion. In particular, disjunctive power for $A$ is not automatically the probability of any rejection in the entire family: a rejection outside $A$ does not satisfy the event. Nor does a false rejection outside $A$ count as detecting a target effect. The tests and their critical values must be specified before either power is calculated.

Joint Distribution of the Contrast Tests

Consider $m$ contrasts estimated under a normal linear model with a common residual variance estimate. For contrast $j$, let $\hat{\theta}_j$ estimate its contrast value $\theta_j$, let $s_j$ be its estimated standard error, and define the signed test statistic $T_j=\hat{\theta}_j/s_j$. Under the model, the test statistics have a joint noncentral t representation. In particular, suppose $Z=(Z_1,\ldots,Z_m)^\mathsf{T}\sim N_m(\delta,R)$, $U\sim\chi^2_v$, and $Z$ and $U$ are independent, where $v$ is the residual degrees of freedom, $R$ is the correlation matrix of the standardized contrast estimates, and $\delta=(\delta_1,\ldots,\delta_m)^\mathsf{T}$ is their vector of signed noncentrality parameters. Then $$ T_j=\frac{Z_j}{\sqrt{U/v}},\qquad j=1,\ldots,m. $$ This representation is exact when the contrast estimates are jointly normal and the common variance estimate is independent of them, as in the normal-theory setting. The shared $U$ is important: even when two contrast estimates are uncorrelated, their t statistics need not be independent.

For a one-sided test in the direction specified by $d_j\in\{-1,1\}$, rejection occurs when $d_jT_j>c_j$, for its chosen critical value $c_j$. For a two-sided test, rejection occurs when $|T_j|>c_j$. Critical values may differ across contrasts. The formulas below use the joint distribution of the statistics and therefore allow correlation and different thresholds.

Theorem: Under the joint representation above, for one-sided tests with rejection events $R_j=\{d_jT_j>c_j\}$, the probability that no contrast in $A$ is rejected is $$ P\left(\bigcap_{j\in A}R_j^c\right) = \int_0^\infty P\left(d_jZ_j\leq c_j\sqrt{u/v}\ \text{for every }j\in A\right) f_{\chi^2_v}(u)\,du. $$ Consequently, disjunctive power is one minus this integral. Conjunctive power is $$ P\left(\bigcap_{j\in A}R_j\right) = \int_0^\infty P\left(d_jZ_j>c_j\sqrt{u/v}\ \text{for every }j\in A\right) f_{\chi^2_v}(u)\,du. $$

Proof. Conditional on $U=u$, the denominator of every statistic is the fixed positive number $\sqrt{u/v}$. Thus $R_j^c$ is equivalent, up to a probability-zero boundary, to $d_jZ_j\leq c_j\sqrt{u/v}$; likewise $R_j$ is equivalent to $d_jZ_j>c_j\sqrt{u/v}$. For any event $B$, the law of total probability gives $P(B)=\int P(B\mid U=u)f_{\chi^2_v}(u)\,du$. Applying this identity to the intersection of the failures, and then to the intersection of the rejections, gives the two formulas. The disjunctive formula follows because the event of at least one rejection is the complement of the event of no rejections. $\square$

The inner probabilities are multivariate normal probabilities with mean $\delta$ and correlation matrix $R$, evaluated over the indicated regions. For two-sided tests, each rejection region has two tails, so the corresponding joint probability is evaluated over the union of those regions; it is not in general a single lower-tail rectangle. These formulations make the role of the dependence explicit and provide a direct route to exact numerical evaluation.

Worked Calculations

Worked Example: Two Orthogonal One-Sided Contrasts

Suppose two prespecified contrasts have orthogonal standardized estimates, so $R$ is the $2$ by $2$ identity matrix. Both are tested in the positive direction at the one-sided critical value $c=t_{0.95,10}$, with $v=10$, and the true noncentrality parameters are $\delta_1=\delta_2=1$. Calculate the disjunctive and conjunctive power for these two targets.

Write $q=\sqrt{u/10}$. Conditional on $U=u$, the two normal variables $Z_1,Z_2$ are independent. For either contrast, $$ P(T_j>c\mid U=u)=P(Z_j>cq)=1-\Phi(cq-1)=\Phi(1-cq). $$ Therefore the conditional probability that both are rejected is $\Phi(1-cq)^2$. Integrating over $U$ gives the exact conjunctive power: $$ \int_0^\infty \Phi\left(1-c\sqrt{u/10}\right)^2 f_{\chi^2_{10}}(u)\,du. $$ The conditional probability that neither is rejected is $\Phi(cq-1)^2$. Hence the exact disjunctive power is $$ 1-\int_0^\infty \Phi\left(c\sqrt{u/10}-1\right)^2 f_{\chi^2_{10}}(u)\,du. $$ These one-dimensional integrals can be evaluated by numerical quadrature. Orthogonality makes the normal numerators independent conditional on $U$, but it does not permit replacing the integrals by products of marginal t-test powers: the same random $U$ appears in both statistics.

Worked Example: Correlated Contrasts With a Known Variance

For a transparent numerical illustration of how dependence changes the two criteria, suppose two one-sided z tests use a known variance, threshold $c=1$, and standardized estimates with a bivariate normal distribution. Let both noncentrality parameters be $1$, and let the correlation be $0$. Each test has power $P(Z_j>1)=1-\Phi(0)=0.5$. Independence gives $$ P(\text{both rejected})=0.5^2=0.25,\qquad P(\text{at least one rejected})=1-(1-0.5)^2=0.75. $$ Now instead suppose the standardized estimates have correlation $1$, so $Z_1=Z_2$ with probability one. Each test still has power $0.5$, but the rejection events coincide. Thus $$ P(\text{both rejected})=0.5,\qquad P(\text{at least one rejected})=0.5. $$ The marginal powers alone did not change; the joint outcomes did. This known-variance example isolates the dependence effect without a shared estimated denominator.

Worked Example: Duplicate t Tests and Target Sets

Suppose a family contains two identical one-sided tests of the same positive contrast, each with $v=10$, critical value $c=t_{0.95,10}$, and true noncentrality parameter $\delta=1$. This is a degenerate but valid joint distribution with $T_1=T_2$. For $U=u$, the probability that either statistic exceeds $c$ is $$ P(Z>c\sqrt{u/10})=1-\Phi\left(c\sqrt{u/10}-1\right). $$ Consequently, the marginal power is $$ \int_0^\infty \left[1-\Phi\left(c\sqrt{u/10}-1\right)\right] f_{\chi^2_{10}}(u)\,du. $$ Because the two rejection events are identical, the probability of at least one rejection and the probability of both rejections are each exactly this same integral. By contrast, if the two tests were independent with this same marginal power $p$, their disjunctive and conjunctive powers would be $1-(1-p)^2$ and $p^2$, respectively. The example shows why power for a set of tests cannot be inferred from its size and marginal powers alone.

Interpreting the Criteria and Avoiding a Common Error

Disjunctive power is useful when success means detecting at least one of several target effects. Conjunctive power is relevant when a claim requires evidence for every specified effect. The distinction is consequential: a procedure can have high probability of finding something while having a much lower probability of finding everything. As the target set grows, disjunctive power cannot decrease, because the event “at least one rejection” becomes larger. Conjunctive power cannot increase, because every added target imposes another rejection requirement.

A common mistake is to multiply marginal powers to get conjunctive power, or to calculate disjunctive power as one minus the product of marginal nonrejection probabilities, without checking independence. Those identities require the rejection events to be independent. Even orthogonal contrast estimates under normal theory can share a variance estimate, inducing dependence among their t statistics. The correction is to use their joint distribution, or condition on the shared variance estimate and then integrate over it, as in the theorem.

A second mistake is to count any family rejection as a successful detection when the stated target set is smaller than the family. Power must match the scientific success event. First specify which contrasts count, and whether success means at least one or all; then calculate that event using the joint distribution and the actual test thresholds.

Check Your Understanding

Use the event definitions and joint-distribution formulas to answer these questions.

For three target contrasts, write the event whose probability is disjunctive power and the event whose probability is conjunctive power.
Why can orthogonal contrast estimates still produce dependent t statistics when they share a residual variance estimate?
If two independent tests each have power $0.8$, calculate their disjunctive and conjunctive powers.
If a rejection outside the specified target set occurs, does it count toward disjunctive power for that set? Explain.
As the target set gains another contrast, what happens to disjunctive and conjunctive power, and why?

Any-Rejection and All-Rejections Power

What You'll Learn