Power Under Orthogonal and Correlated Contrasts

Why Dependence Matters for Power

When several contrasts are tested together, the probability of detecting at least one nonzero effect depends on more than the separate power of each test. The estimates may move together or in opposite directions, so their rejection events overlap differently. This tutorial calculates that familywise rejection probability under specified alternatives, first for joint normal statistics and then for statistics with a shared variance estimate.

The critical values are treated as given in the calculations below. This separates the effect of dependence under the alternative from the separate problem of choosing a critical value to control familywise error. In an application, the critical value should match the procedure and error criterion being used.

Definition: For $m$ specified contrast tests, familywise power at an alternative is the probability that at least one test rejects its null hypothesis. If $R_j$ is the event that test $j$ rejects, then $$ \operatorname{Power}_{\mathrm{FW}}=P\left(\bigcup_{j=1}^{m}R_j\right) =1-P\left(\bigcap_{j=1}^{m}R_j^c\right). $$ This is a probability under the specified alternative, not under the joint null.

The complement form is especially useful: calculate the probability that all tests fail to reject, then subtract it from one. It also makes clear why separate marginal powers are not enough. The probability of the intersection of the non-rejection events depends on the joint distribution.

Covariance of Contrast Estimates

Suppose $Y$ is a jointly normal vector with mean vector $\mu$ and covariance matrix $\Sigma$. For two contrast coefficient vectors $a$ and $b$, the estimates $a^\mathsf{T}Y$ and $b^\mathsf{T}Y$ have covariance $a^\mathsf{T}\Sigma b$. Their standardized versions therefore have a correlation determined by both the contrast vectors and the covariance structure of the data.

Theorem: If $Y\sim N(\mu,\Sigma)$, then $$ \begin{pmatrix}a^\mathsf{T}Y\\ b^\mathsf{T}Y\end{pmatrix} \sim N\left[ \begin{pmatrix}a^\mathsf{T}\mu\\ b^\mathsf{T}\mu\end{pmatrix}, \begin{pmatrix} a^\mathsf{T}\Sigma a & a^\mathsf{T}\Sigma b\\ b^\mathsf{T}\Sigma a & b^\mathsf{T}\Sigma b \end{pmatrix} \right]. $$ If both variances are positive, the correlation is $$ \rho=\frac{a^\mathsf{T}\Sigma b} {\sqrt{(a^\mathsf{T}\Sigma a)(b^\mathsf{T}\Sigma b)}}. $$ In particular, the two estimates are independent if $a^\mathsf{T}\Sigma b=0$.

To verify the result, any linear combination $s\,a^\mathsf{T}Y+t\,b^\mathsf{T}Y$ equals $(sa+tb)^\mathsf{T}Y$, which is normal because $Y$ is jointly normal. Its mean is $s\,a^\mathsf{T}\mu+t\,b^\mathsf{T}\mu$, and its variance is $s^2a^\mathsf{T}\Sigma a+2st\,a^\mathsf{T}\Sigma b+t^2b^\mathsf{T}\Sigma b$. These are precisely the mean and covariance specifications displayed above. A jointly normal pair with zero covariance has independent components, proving the final claim. $\square$

The result identifies orthogonality in the relevant statistical metric: it is the covariance $a^\mathsf{T}\Sigma b$, not merely a visual or informal notion of contrast separation, that determines whether the estimates are uncorrelated. Under joint normality, zero covariance also gives independence.

Worked Example: Power for Two Orthogonal Contrasts

Suppose two standardized contrast estimates have independent normal distributions with unit variances and alternative means $\delta_1=0$ and $\delta_2=\sqrt{2/3}$. Each test rejects when the absolute standardized statistic exceeds $c=1.96$. Find the familywise power.

For a statistic $Z\sim N(\delta,1)$, the probability of not rejecting a two-sided test is $P(-c\leq Z\leq c)=\Phi(c-\delta)-\Phi(-c-\delta)$, where $\Phi$ is the standard normal distribution function. Thus the first test's non-rejection probability is $\Phi(1.96)-\Phi(-1.96)$. The second test's is $\Phi(1.96-\sqrt{2/3})-\Phi(-1.96-\sqrt{2/3})$.

The statistics are independent, so their non-rejection events are independent. Therefore $$ \operatorname{Power}_{\mathrm{FW}} =1- [\Phi(1.96)-\Phi(-1.96)] [\Phi(1.96-\sqrt{2/3})-\Phi(-1.96-\sqrt{2/3})]. $$

This is an exact expression for the familywise power at the specified alternative and cutoff. The first contrast has no mean shift, but it still contributes a chance of rejection; the second has nonzero shift. The calculation combines both chances through the product of their non-rejection probabilities, rather than adding their marginal powers.

Correlated Normal Contrasts: A Bivariate Calculation

For correlated standardized statistics $Z_1,Z_2$ with unit variances, means $\delta_1,\delta_2$, and correlation $\rho$, the familywise power for two-sided tests with common cutoff $c$ is $$ 1-P(-c\leq Z_1\leq c,\ -c\leq Z_2\leq c). $$ A convenient way to calculate the rectangle probability is to condition on $Z_1=x$. The conditional distribution is $Z_2\mid Z_1=x\sim N(\delta_2+\rho(x-\delta_1),1-\rho^2)$, provided $|\rho|<1$. Integrating the conditional probability over the non-rejection range for $Z_1$ gives a one-dimensional integral.

Theorem: If $(Z_1,Z_2)$ is jointly normal with means $(\delta_1,\delta_2)$, unit variances and correlation $|\rho|<1$, then for common two-sided cutoff $c>0$, $$ \operatorname{Power}_{\mathrm{FW}} =1-\int_{-c}^{c}\phi(x-\delta_1) \left[ \Phi\left(\frac{c-\delta_2-\rho(x-\delta_1)}{\sqrt{1-\rho^2}}\right) -\Phi\left(\frac{-c-\delta_2-\rho(x-\delta_1)}{\sqrt{1-\rho^2}}\right) \right]\,dx, $$ where $\phi$ and $\Phi$ are the standard normal density and distribution function.

For the proof, the joint non-rejection probability is $P(-c\leq Z_1\leq c,\,-c\leq Z_2\leq c)$. The density of $Z_1$ is $\phi(x-\delta_1)$. Given $Z_1=x$, the probability that $Z_2$ lies between $-c$ and $c$ is the difference of the two conditional normal distribution functions in the integrand. Integrating this conditional probability against the density of $Z_1$ over $[-c,c]$ gives the joint non-rejection probability. Taking its complement proves the formula. $\square$

Worked Example: Correlation Changes a Two-Contrast Calculation

Let $(Z_1,Z_2)$ be jointly normal with alternative mean $(1,1)$, unit marginal variances and correlation $1/2$. Both tests use cutoff $c=2$. Calculate the familywise power as an exact integral.

Conditioning on $Z_1=x$, the conditional mean of $Z_2$ is $1+\tfrac12(x-1)=\tfrac12x+\tfrac12$, and its conditional standard deviation is $\sqrt{1-(1/2)^2}=\sqrt{3}/2$. Thus the conditional probability of no rejection by the second test is $$ \Phi\left(\frac{2-(\tfrac12x+\tfrac12)}{\sqrt{3}/2}\right) -\Phi\left(\frac{-2-(\tfrac12x+\tfrac12)}{\sqrt{3}/2}\right) = \Phi\left(\frac{3-x}{\sqrt{3}}\right) -\Phi\left(\frac{-5-x}{\sqrt{3}}\right). $$

The lower bound follows directly from $-2-(\tfrac12x+\tfrac12)=-\tfrac52-\tfrac12x$; dividing by $\sqrt{3}/2$ gives $(-5-x)/\sqrt{3}$. The first statistic has density $\phi(x-1)$. Hence the exact familywise power is $$ 1-\int_{-2}^{2}\phi(x-1) \left[ \Phi\left(\frac{3-x}{\sqrt{3}}\right) -\Phi\left(\frac{-5-x}{\sqrt{3}}\right) \right]\,dx. $$

The integral is the joint probability that neither statistic crosses its cutoff, and the leading one subtracts that probability from one. This expression explicitly retains the dependence; replacing it by a product of two marginal non-rejection probabilities would be incorrect because the correlation is $1/2$, not zero.

Shared Variance Estimates and Multivariate t Power

In normal-theory inference with an unknown common variance, standardized contrast statistics often share the same estimated variance. Their joint distribution is then a multivariate t distribution, not a collection of independent univariate t variables. Even orthogonal contrast numerators may yield dependent t statistics because their denominators contain the same random scale.

Definition: A noncentral multivariate t vector with degrees of freedom $\nu$, correlation matrix $R$, and noncentrality vector $\delta$ can be represented as $$ T=\frac{Z+\delta}{\sqrt{U/\nu}}, $$ where $Z\sim N(0,R)$, $U\sim\chi^2_\nu$, and $Z$ and $U$ are independent. The division by the scalar applies to every coordinate.

This representation gives an exact route to power: condition on $U=u$, calculate a multivariate normal non-rejection probability with cutoffs scaled by $\sqrt{u/\nu}$, and average over the chi-square distribution. The shared scale must be kept common across coordinates. Treating each t statistic as if it had an independent denominator changes the joint distribution.

Worked Example: Two Orthogonal Numerators with a Common t Denominator

Suppose $T_1,T_2$ have the representation above with $\nu=4$, $R$ equal to the $2$ by $2$ identity matrix, noncentrality vector $(1,0)$, and common two-sided cutoff $c=2.5$. Find the exact familywise power.

Conditional on $U=u$, neither test rejects precisely when $-2.5\sqrt{u/4}\leq Z_1+1\leq2.5\sqrt{u/4}$ and $-2.5\sqrt{u/4}\leq Z_2\leq2.5\sqrt{u/4}$. The two normal coordinates are independent conditional on $u$. Their respective probabilities are $$ \Phi(2.5\sqrt{u/4}-1)-\Phi(-2.5\sqrt{u/4}-1) $$ and $$ \Phi(2.5\sqrt{u/4})-\Phi(-2.5\sqrt{u/4}). $$

Averaging their product over $U\sim\chi^2_4$ and taking the complement gives $$ 1-\int_0^\infty f_{\chi^2_4}(u) [\Phi(2.5\sqrt{u/4}-1)-\Phi(-2.5\sqrt{u/4}-1)] [\Phi(2.5\sqrt{u/4})-\Phi(-2.5\sqrt{u/4})]\,du. $$

Although the normal numerators are independent, the t statistics are not independent: both are divided by $\sqrt{U/4}$. The product inside the integral is valid conditional on $U=u$, while the integral correctly averages over their shared random denominator.

How to Interpret Comparisons

For a fixed cutoff, a correlation changes the joint non-rejection probability and therefore changes familywise power. There is no general rule that positive correlation always increases or always decreases power. The result depends on the alternative mean vector as well as the correlation: shifting the means along a direction of strong common variation has a different effect from shifting them in opposing directions.

A further distinction is important. If a procedure chooses its critical value from the joint null distribution, changing the correlation can change both the alternative rejection probability and the critical value. A comparison at a fixed cutoff isolates the first effect; a comparison of complete procedures must use each procedure's own calibrated cutoff. Neither comparison can be replaced by comparing marginal powers alone.

Common Mistakes

Multiplying marginal non-rejection probabilities when tests are correlated. The product is valid only when the rejection events are independent. Use the joint distribution or condition on one statistic.
Assuming orthogonal numerators make t statistics independent. With a shared variance estimate, the common random denominator induces dependence even when the normal numerators are independent.
Using the null distribution to describe power. Power is evaluated under a specified alternative, so the mean vector or noncentrality parameters must be included.
Changing the critical value without saying so. A fixed-cutoff comparison isolates dependence under the alternative; a calibrated-procedure comparison also reflects the dependence of the null critical value on the joint distribution.

Key takeaway: Familywise power is one minus the joint probability of no rejections. Orthogonal normal contrasts permit a product calculation, but correlated contrasts require their joint distribution, and a shared variance estimate must be handled as a common random scale.

Check Your Understanding

Use the joint distribution, not marginal powers alone, to answer these questions.

For jointly normal contrast estimates, what covariance expression determines whether two estimates are independent?
Why is the familywise power for independent tests calculated from the product of non-rejection probabilities?
In the correlated example, derive the conditional mean and variance of $Z_2$ given $Z_1=x$.
Why does a common variance estimate make orthogonal t statistics dependent?
What additional issue must be considered when comparing procedures that calibrate different critical values from their joint null distributions?