Power for Families of Planned Multivariate Contrasts

What Does It Mean to Detect a Family of Contrasts?

A planned multivariate comparison concerns a vector of response means, not just one scalar outcome. If there are $q$ groups and each observation has $p$ response components, write the group means as $\mu_1,\ldots,\mu_q$, where each $\mu_i$ is a $p$-vector. A planned contrast $j$ has coefficients $c_{j1},\ldots,c_{jq}$, summing to zero, and its target is the vector $\delta_j=\sum_i c_{ji}\mu_i$. Its null hypothesis is $H_{0j}:\delta_j=0$.

For a family of $m$ planned contrasts, familywise power will mean the probability of rejecting at least one null hypothesis that is false. This is often called disjunctive power. It differs from the probability of rejecting every false null, sometimes called conjunctive power. The distinction matters: a procedure can have a good chance of finding at least one real effect while still having a lower chance of finding all of them.

Definition: Let $J_1=\{j:\delta_j\ne 0\}$ be the set of contrasts with nonzero effects. The familywise power for detecting at least one specified effect is $$ P\left(\bigcup_{j\in J_1}\{\text{the procedure rejects }H_{0j}\}\right). $$ It depends both on the effects and on the joint distribution of the contrast estimates, as well as on the familywise testing procedure.

The calculations below use independent Gaussian groups with a common positive-definite covariance matrix $\Sigma$. This setting gives exact marginal Hotelling tests and a direct route to joint-power simulation. Unequal-covariance procedures can also be evaluated by the same general principle—compute or simulate the joint rejection event—but their reference distributions and covariance estimates must match the procedure being used.

How Contrast Estimates Depend on One Another

Let group $i$ have sample size $n_i$, sample mean $\bar X_i$, and population mean $\mu_i$. The estimate of contrast $j$ is $\hat\delta_j=\sum_i c_{ji}\bar X_i$. Under the common-covariance model, the group sample means are independent and $\operatorname{Cov}(\bar X_i)=\Sigma/n_i$.

Result: For two planned contrasts $j$ and $k$, $$ E[\hat\delta_j]=\delta_j,\qquad \operatorname{Cov}(\hat\delta_j,\hat\delta_k) =\left(\sum_{i=1}^q\frac{c_{ji}c_{ki}}{n_i}\right)\Sigma. $$ In particular, define $v_j=\sum_i c_{ji}^2/n_i$. Then $\operatorname{Var}(\hat\delta_j)=v_j\Sigma$.

To derive the covariance, expand both estimates as sums of group sample means. Terms from different groups have zero covariance by independence, leaving $$ \operatorname{Cov}(\hat\delta_j,\hat\delta_k) =\sum_{i=1}^q c_{ji}c_{ki}\operatorname{Cov}(\bar X_i) =\left(\sum_{i=1}^q\frac{c_{ji}c_{ki}}{n_i}\right)\Sigma. $$ Linearity of expectation gives $E[\hat\delta_j]=\sum_i c_{ji}\mu_i=\delta_j$, and setting $k=j$ gives the stated variance. $\square$

The scalar coefficient in this covariance formula captures the overlap between the two contrast patterns, adjusted for group sample sizes. If it is zero, the estimates are uncorrelated; because they are jointly Gaussian under this model, they are then independent. If it is nonzero, treating the tests as independent can misstate familywise power.

Worked Example: Finding the Covariance Between Two Contrasts

There are three independent groups of 10 observations each. Two planned contrasts have coefficient vectors $c_1=(1,-1,0)$ and $c_2=(1,0,-1)$. Each response has covariance matrix $\Sigma=I_2$, the $2\times2$ identity matrix.

For either contrast, $v_j=(1^2+(-1)^2)/10=0.2$. Their cross-covariance coefficient is $$ \frac{1\cdot1+(-1)\cdot0+0\cdot(-1)}{10}=0.1. $$ Thus $\operatorname{Var}(\hat\delta_1)=\operatorname{Var}(\hat\delta_2)=0.2I_2$, and $\operatorname{Cov}(\hat\delta_1,\hat\delta_2)=0.1I_2$. The correlation between corresponding coordinates of the two estimates is $0.1/\sqrt{0.2\cdot0.2}=0.5$. The estimates are not independent, so multiplying their nonrejection probabilities would not give the familywise power.

Exact Marginal Power and a Familywise Adjustment

Suppose the groups are Gaussian with common covariance $\Sigma$, and let $S$ be the pooled within-group sample covariance with $\nu=N-q$ degrees of freedom, where $N=\sum_i n_i$. For contrast $j$, its Hotelling statistic is $$ T_j^2=\frac{\hat\delta_j^{\mathsf T}S^{-1}\hat\delta_j}{v_j}. $$ Under $H_{0j}$, $\frac{\nu-p+1}{p\nu}T_j^2$ has an $F$ distribution with $p$ and $\nu-p+1$ degrees of freedom. Under a fixed alternative, it has a noncentral $F$ distribution with the same degrees of freedom and noncentrality parameter $$ \lambda_j=\frac{\delta_j^{\mathsf T}\Sigma^{-1}\delta_j}{v_j}. $$ This parameter measures the contrast effect in covariance-adjusted units, relative to the sampling variance of its estimate.

If Bonferroni testing is used for $m$ contrasts at familywise level $\alpha$, each test uses level $\alpha/m$. Let $f_{p,\nu-p+1;1-\alpha/m}$ denote the upper $1-\alpha/m$ quantile of the central $F$ distribution with the indicated degrees of freedom. The rejection rule for contrast $j$ is $$ \frac{\nu-p+1}{p\nu}T_j^2>f_{p,\nu-p+1;1-\alpha/m}. $$ Its marginal power is the upper-tail probability of the noncentral $F$ distribution at this cutoff. This is an exact marginal calculation under the stated model; it does not, by itself, give the probability of detecting at least one effect in the family.

For a given procedure, write $A_j$ for the event that it rejects contrast $j$. Familywise power is $P(\bigcup_{j\in J_1}A_j)$, not the sum of the marginal powers. When the events are independent, the union probability is $1-\prod_{j\in J_1}(1-P(A_j))$. Without independence, that product formula is not valid. A useful general bound is $$ \max_{j\in J_1}P(A_j)\ \leq\ P\left(\bigcup_{j\in J_1}A_j\right)\ \leq\ \min\left(1,\sum_{j\in J_1}P(A_j)\right). $$ The lower bound holds because the union contains every event; the upper bound is the union bound.

Worked Example: Approximate Familywise Power for Orthogonal Contrasts

Consider four groups of 10 observations each, two response components with $\Sigma=I_2$, and two contrasts $c_1=(1,-1,0,0)$ and $c_2=(0,0,1,-1)$. Both have $v_j=0.2$, while their cross-covariance coefficient is zero. Suppose each true contrast is $\delta_j=(\sqrt{0.8},0)$. Then $\lambda_j=\delta_j^{\mathsf T}\delta_j/0.2=4$.

For illustration, use the large-sample approximation that replaces the estimated covariance by the known $\Sigma$. Each statistic is then a squared norm of a two-dimensional normal vector, and its alternative distribution is noncentral chi-square with 2 degrees of freedom and noncentrality 4. Bonferroni testing at familywise level $0.05$ uses per-test level $0.025$; the upper-tail chi-square cutoff is approximately $7.378$.

For $x=7.378$, the noncentral chi-square survival probability can be calculated by the Poisson-mixture identity $$ P(\chi^2_2(4)>x) =\sum_{k=0}^{\infty}P(K=k)\,P(\chi^2_{2+2k}>x),\qquad K\sim\operatorname{Poisson}(2). $$ For example, the first terms use Poisson weights $0.1353, 0.2707, 0.2707,\ldots$ and central chi-square upper tails approximately $0.025, 0.117, 0.287,\ldots$. Summing the terms gives marginal power of about $0.31$. In this known-covariance approximation the two estimates are independent, so familywise power is approximately $1-(1-0.31)^2=0.524$. With an estimated pooled covariance, the individual tests share $S$; exact finite-sample joint power should instead account for that shared estimate.

Joint Power When the Tests Are Dependent

For planned contrasts with nonzero covariance, joint power is most directly obtained from the joint sampling distribution. In the common-covariance Gaussian model, the vector of all contrast estimates is jointly normal, with means $\delta_j$ and covariance blocks given by the formula above. A large-sample approximation replaces $S$ by $\Sigma$ and evaluates the rejection regions using this joint normal distribution. For small or moderate samples, simulation can retain the random pooled covariance and the exact finite-sample test thresholds.

Specify the effects and design.
Give the group means, sample sizes, common covariance matrix, and every planned contrast. Identify which contrasts are nonzero.

Generate a replicate under the alternative.
Generate independent Gaussian group sample means with means $\mu_i$ and covariances $\Sigma/n_i$. For an exact pooled-Hotelling calculation, also generate an independent pooled covariance $S$ with $\nu S$ having a Wishart distribution with scale $\Sigma$ and $\nu$ degrees of freedom.

Apply the entire familywise procedure.
Form every $\hat\delta_j$, calculate its test statistic, apply the chosen adjusted cutoff, and record whether at least one false null was rejected.

Estimate power and its simulation error.
Repeat for $B$ independent replicates. If $R$ replicates detect at least one specified effect, estimate familywise power by $R/B$; its Monte Carlo standard error is approximately $\sqrt{(R/B)(1-R/B)/B}$.

The simulation must reproduce the procedure actually intended for use. For example, a Bonferroni procedure uses its per-test adjusted thresholds in every replicate; a different familywise method requires its own rejection rule. Replacing all tests by independent marginal draws discards the dependence that the contrast coefficients and shared covariance estimate create.

Worked Example: Setting Up a Dependent-Family Calculation

Return to the three-group design with $c_1=(1,-1,0)$, $c_2=(1,0,-1)$, $n_i=10$, and $\Sigma=I_2$. Suppose the specified true effects are $\delta_1=(0.6,0)$ and $\delta_2=(0,0.6)$. For each contrast, $v_j=0.2$, so both noncentrality parameters are $0.6^2/0.2=1.8$. The standardized contrast estimates have correlation $0.5$ between corresponding response coordinates.

Under a known-covariance approximation, define $Z_j=\hat\delta_j/\sqrt{0.2}$. Their means are $(0.6/\sqrt{0.2},0)$ and $(0,0.6/\sqrt{0.2})$, and each has covariance $I_2$. Their cross-covariance matrix is $0.5I_2$. With two Bonferroni tests at familywise level $0.05$, the rejection event is $\|Z_1\|^2>7.378$ or $\|Z_2\|^2>7.378$. The desired power is therefore the probability of this union under the joint four-dimensional normal distribution with the stated means and covariance. It can be evaluated by numerical integration or estimated by drawing from that joint normal distribution and counting union rejections. Using $1-(1-\pi_1)(1-\pi_2)$ would assume independence and is unjustified here.

Common Pitfall and Practical Interpretation

A frequent mistake is to calculate each contrast’s power and then combine the results as though the tests were independent. Even if every contrast uses a valid marginal test, shared groups can make the contrast estimates correlated; estimated covariance matrices can also induce dependence among test statistics. The correction is to derive the contrast covariance from the coefficients and sample sizes, then compute the joint rejection probability using that dependence. If a closed-form joint probability is unavailable, simulation is a principled approximation.

Power also depends on the familywise procedure. A more stringent adjusted cutoff generally lowers the probability of rejection for each contrast, while dependence determines how those marginal probabilities combine into a familywise probability. Report which contrasts are assumed to be nonzero and whether the target is detecting at least one or detecting all of them. That makes the power statement interpretable rather than merely a list of marginal calculations.

Key takeaway: Marginal power is governed by each contrast’s covariance-adjusted effect and test cutoff. Familywise power additionally depends on the joint distribution of all planned contrast estimates and on the exact rejection rule used for the family.

Check Your Understanding

Use the definitions and results above to answer these questions.

What is the difference between disjunctive familywise power and the probability of detecting every false null?
For two contrasts, what quantity determines whether their estimates are uncorrelated under the common-covariance model?
What is the noncentrality parameter for a planned multivariate contrast under the pooled Hotelling model?
When is it valid to combine marginal powers using $1-\prod_j(1-\pi_j)$?
Why must a joint-power simulation apply the adjusted rejection rule to all contrasts in each replicate?