James-Type Tests for Several Means With Unequal Covariances

Why a Several-Group Test Needs an Approximation

When independent groups have different covariance matrices, the pooled-covariance Hotelling test does not match the sampling structure: there is no single covariance matrix shared by the groups. A natural alternative is to compare the estimated mean contrasts using their estimated covariance. The resulting quadratic form is a Wald statistic, but its reference distribution is not generally an exact F distribution because the covariance estimate combines contributions from several group-specific sample covariance matrices.

A James-type test addresses this calibration problem by approximating the covariance estimate with a scaled Wishart matrix. Moment matching supplies an effective degrees of freedom, and the resulting Wald statistic is compared with an F distribution. This is an approximation, not an exact finite-sample calibration.

Setup and definition: There are $k$ independent groups. Group $i$ has $n_i$ observations on a $p$-dimensional random vector, with population mean $\mu_i$ and covariance matrix $\Sigma_i$. The covariance matrices may differ across groups. The hypothesis is $H_0:\mu_1=\cdots=\mu_k$. Write $\bar X_i$ and $S_i$ for the sample mean vector and unbiased sample covariance matrix in group $i$. Assume independent multivariate normal observations within each group, with $n_i\geq 2$.

Contrasts and the James-Type Statistic

Stack the group means into $\bar X=(\bar X_1',\ldots,\bar X_k')'$. Choose a $q\times kp$ contrast matrix $C=(C_1,\ldots,C_k)$ of full row rank, where each $C_i$ has $p$ columns. For the equality hypothesis, take $q=p(k-1)$ and choose $C$ so that $C\mu=0$ holds exactly when all group means are equal. For example, contrasts of each of the first $k-1$ means with the last group give such a matrix. The contrast estimate is $Y=C\bar X=\sum_i C_i\bar X_i$.

Under the null, $E[Y]=0$. Independence between groups gives the covariance of $Y$ as a sum of group contributions. Estimate each contribution using that group's own sample covariance:

\widehat\Omega=\sum_{i=1}^k B_i,\qquad B_i=C_i\frac{S_i}{n_i}C_i',\qquad Q=Y'\widehat\Omega^{-1}Y.

Here $Q$ is the estimated-covariance Wald quadratic form. It has $q$ contrast dimensions. To calibrate it, define $A_i=\widehat\Omega^{-1/2}B_i\widehat\Omega^{-1/2}$, where the inverse square root exists when $\widehat\Omega$ is positive definite. The moment-matched degrees of freedom are

\widehat\nu= \frac{q(q+1)} {\displaystyle\sum_{i=1}^k \frac{\{\operatorname{tr}(A_i)\}^2+\operatorname{tr}(A_i^2)} {n_i-1}}.

Equivalently, the traces can be computed without explicitly forming the inverse square root by using $\widehat\Omega^{-1}B_i$ in their place. The James-type statistic and its reference distribution are

F_J=\frac{\widehat\nu-q+1}{\widehat\nu q}\,Q \quad\text{and}\quad F_J\ \text{is compared approximately with }\ F_{q,\,\widehat\nu-q+1}.

The denominator degrees of freedom must be positive, so the calculation requires $\widehat\nu>q-1$. In practice, the contrast covariance estimate must also be nonsingular. A small sample or nearly singular covariance estimate can make the approximation unreliable even when the formula is computable.

Why the Degrees-of-Freedom Formula Has This Form

The formula matches the overall covariance variability of the estimated contrast covariance to that of a Wishart covariance estimate. The derivation below assumes the stated Gaussian model, so that each $(n_i-1)S_i$ has a Wishart distribution and is independent of the group sample mean. This also makes $Y$ independent of $\widehat\Omega$.

Moment-matching result: Let $\Omega=\operatorname{Var}(Y)$ under $H_0$, and let $B_i^*=C_i(\Sigma_i/n_i)C_i'$, so that $\Omega=\sum_i B_i^*$. For Gaussian groups, the covariance variability of $\widehat\Omega$, after whitening by $\Omega$, has total elementwise variance $$ \sum_{a,b=1}^q \operatorname{Var}\!\left( [\Omega^{-1/2}\widehat\Omega\Omega^{-1/2}]_{ab} \right) = \sum_{i=1}^k \frac{\{\operatorname{tr}(A_i^*)\}^2+\operatorname{tr}((A_i^*)^2)} {n_i-1}, \qquad A_i^*=\Omega^{-1/2}B_i^*\Omega^{-1/2}. $$ A scaled Wishart covariance estimate with degrees of freedom $\nu$ has this total equal to $q(q+1)/\nu$. Equating the two totals gives the stated degrees-of-freedom formula, with unknown population quantities replaced by estimates.

To verify the result, put $E_i=C_iS_iC_i'/n_i$. For a sample covariance from $n_i$ independent Gaussian observations, the Wishart covariance identity gives $\operatorname{Cov}((S_i)_{ab},(S_i)_{cd})= \{(\Sigma_i)_{ac}(\Sigma_i)_{bd}+(\Sigma_i)_{ad}(\Sigma_i)_{bc}\}/(n_i-1)$. A linear transformation preserves the corresponding Wishart covariance identity. Thus, after whitening, the contribution from group $i$ has total elementwise variance $\{(\operatorname{tr}A_i^*)^2+\operatorname{tr}((A_i^*)^2)\}/(n_i-1)$. The groups are independent, so these contributions add. For a whitened Wishart matrix with mean $I_q$ and degrees of freedom $\nu$, the variance of element $(a,b)$ is $(\delta_{ac}\delta_{bd}+\delta_{ad}\delta_{bc})/\nu$ in covariance notation; summing the diagonal element variances and twice the off-diagonal variances gives $q(q+1)/\nu$. Equating totals proves the moment-matching expression. Replacing population matrices by their sample estimates yields $\widehat\nu$. $\square$

The F conversion follows from the corresponding Wishart identity. If $Z\sim N_q(0,I_q)$ and independently $W\sim\operatorname{Wishart}_q(I_q,\nu)$, then $Z'(W/\nu)^{-1}Z=\nu Z'W^{-1}Z$. Integrating the independent normal and Wishart densities gives the density of $R=Z'W^{-1}Z$:

f_R(r)= \frac{\Gamma((\nu+1)/2)} {\Gamma(q/2)\Gamma((\nu-q+1)/2)} r^{q/2-1}(1+r)^{-(\nu+1)/2}, \qquad r>0.

This is the beta-prime density of $(\chi_q^2)/(\chi_{\nu-q+1}^2)$, with the two chi-square variables independent. Therefore $R$ has distribution $qF_{q,\nu-q+1}/(\nu-q+1)$. Under the scaled-Wishart approximation to $\widehat\Omega$, the same identity yields $(\widehat\nu-q+1)Q/(\widehat\nu q)$ as an approximate $F_{q,\widehat\nu-q+1}$ statistic.

Worked Calculations

Worked Example: Encoding Equality of Three Mean Vectors

Suppose three independent groups each have $p=2$ measurements. Equality of their mean vectors imposes four scalar contrast restrictions. Compare groups 1 and 2 with group 3 by taking $C_1=\begin{pmatrix}1&0\\0&1\\0&0\\0&0\end{pmatrix}$, $C_2=\begin{pmatrix}0&0\\0&0\\1&0\\0&1\end{pmatrix}$, and $C_3=\begin{pmatrix}-1&0\\0&-1\\-1&0\\0&-1\end{pmatrix}$.

Then $q=4$ and $Y=(\bar X_1-\bar X_3,\bar X_2-\bar X_3)'$, with each two-dimensional difference stacked as a block. The equation $C\mu=0$ says $\mu_1-\mu_3=0$ and $\mu_2-\mu_3=0$, which is precisely equality of all three mean vectors. The covariance estimate is formed by adding $C_i(S_i/n_i)C_i'$ for all three groups. Group 3 contributes to both contrasts, so its contribution includes covariance between those two contrast blocks; it must not be omitted or counted as if the contrasts were independent.

Worked Example: Applying the James-Type F Calibration

For a three-group, two-dimensional comparison, suppose $q=4$, the computed Wald statistic is $Q=1.2$, and the three contributions to the denominator of the degrees-of-freedom formula are $0.4000$, $0.3000$, and $0.2066$. These contributions are each the trace expression for one group divided by its $n_i-1$. Their sum is $0.9066$, while $q(q+1)=4(5)=20$. Thus

\widehat\nu=\frac{20}{0.9066}=22.0604\ldots\approx22.06.

The numerator degrees of freedom are $q=4$, and the approximate denominator degrees of freedom are $22.06-4+1=19.06$. Substitution gives

F_J= \frac{(22.06-4+1)(1.2)}{22.06(4)} = \frac{19.06(1.2)}{88.24} = \frac{22.872}{88.24} \approx0.2592.

The test compares $0.2592$ with an $F_{4,19.06}$ reference distribution. This is far below the usual 5% upper-tail critical value, which is about 2.9, so the null hypothesis is not rejected at level 0.05. The conclusion is a lack of evidence against equality, not proof that the population mean vectors are identical.

Worked Example: Reading the Calibration Before Testing

Consider a second four-dimensional contrast problem with $\widehat\Omega=\operatorname{diag}(2,1,1,0.5)$ and $Y=(1,1,0,0)'$. Then the inverse covariance estimate is $\operatorname{diag}(0.5,1,1,2)$, and

Q=Y'\widehat\Omega^{-1}Y =(1)(0.5)(1)+(1)(1)(1)=1.5.

Suppose the group trace contributions in the degrees-of-freedom formula sum to $0.5$. Since $q(q+1)=20$, the matched degrees of freedom are $\widehat\nu=20/0.5=40$. The statistic is therefore

F_J=\frac{40-4+1}{40(4)}(1.5) =\frac{37(1.5)}{160} =0.346875, \qquad F_J\ \text{is compared with }F_{4,37}.

This example separates two inputs that are sometimes confused: $Q$ measures the observed contrast relative to its estimated covariance, while $\widehat\nu$ describes the estimated covariance's variability. A larger $Q$ does not itself imply a particular denominator degrees of freedom.

Assumptions, Interpretation, and a Common Pitfall

The Gaussian assumption provides the Wishart covariance identity and independence of each sample mean from its sample covariance. Independent groups are also essential: dependence between groups changes the covariance of the stacked contrasts and invalidates the sum-of-group-contributions construction as written. In applications with substantial nonnormality, small samples, or highly unstable covariance estimates, the nominal F calibration can be inaccurate. The statistic should be reported as an approximate test, alongside the group sample sizes and the covariance-estimation method.

A frequent mistake is to use the raw quadratic form $Q$ as though it were itself an F statistic with $q$ numerator degrees of freedom, or to use the pooled two-sample calibration despite unequal group covariances. The correction is to compute the moment-matched $\widehat\nu$, use the factor $(\widehat\nu-q+1)/(\widehat\nu q)$, and state the reference distribution $F_{q,\widehat\nu-q+1}$. Another mistake is to assume that an approximate F reference makes the test exact. The calibration rests on a scaled-Wishart moment match; it does not turn the heterogeneous-covariance problem into an exact pooled-covariance test.

Key takeaway: For equality of several independent group mean vectors, form a full-rank contrast vector, estimate its covariance using each group's own covariance matrix, and calibrate the Wald quadratic form with the James-type moment-matched F approximation. Check positive definiteness and the estimated degrees of freedom before interpreting the result.

Check Your Understanding

Use the definitions and calculations above to answer these questions.

For $k$ groups with $p$-dimensional means, how many independent scalar restrictions express equality of all group means?
Why does the covariance contribution from a group-specific sample covariance have $n_i-1$ in the degrees-of-freedom calculation?
If $q=3$, $Q=2$, and $\widehat\nu=18$, write the James-type F statistic and its reference degrees of freedom.
Why is the James-type calibration approximate rather than an exact finite-sample F test?
What problem arises if the estimated contrast covariance $\widehat\Omega$ is singular?