Exact F Calibration for the Pooled Two-Sample Test

Why the Calibration Is Exact

When two independent groups are measured on several variables, the mean difference is a vector, and its uncertainty depends on the covariance among those variables. The pooled two-sample Hotelling statistic combines the observed difference with an estimate of that shared covariance. Under the right sampling model, its null distribution is not merely approximated by an F distribution: a scaled version has an exact F distribution in finite samples.

The exact calibration is useful because it turns a multivariate distance into a test with explicit degrees of freedom. It also makes the assumptions visible. In particular, the result relies on independent multivariate normal samples and a common positive-definite covariance matrix. A similar-looking statistic under other conditions need not have this exact null distribution.

Statistic and Assumptions

Let \(X_{1,1},\ldots,X_{1,n_1}\) and \(X_{2,1},\ldots,X_{2,n_2}\) be independent samples of \(p\)-dimensional random vectors. Assume that observations within each group are identically distributed and that \(X_{i,j}\sim N_p(\mu_i,\Sigma)\) for \(i=1,2\), where \(\Sigma\) is the same positive-definite covariance matrix in both groups. Write \(N=n_1+n_2\), \(\nu=N-2\), and let \(\bar X_1,\bar X_2\) and \(S_1,S_2\) denote the group sample means and sample covariance matrices.

For the stated normal model, the pooled residual degrees of freedom are \(\nu\). The pooled covariance is invertible with probability one when \(\nu\geq p\). Thus, the exact result below requires \(n_1,n_2\geq1\) and \(\nu\geq p\); the second F degree of freedom, \(\nu-p+1\), is then positive. In practical applications each group will generally have several observations so its covariance can be estimated, but the pooled residual sum of squares can have the stated distribution even if one group contributes zero residual degrees of freedom.

The Exact F Result

The numerator degrees of freedom are the dimension \(p\) of the mean-difference vector. The denominator degrees of freedom are reduced from the pooled residual degrees of freedom \(\nu\) by \(p-1\). The reduction reflects that the covariance matrix is estimated from the same data used in the test.

Here is a derivation. Under the normal model, the sample mean in each group is independent of that group’s centered residual sum of squares. The two group samples are independent, so the difference of their means is also independent of the pooled residual sum of squares. Moreover, the sum of the two independent residual sums of squares has a Wishart distribution. Define \(a=n_1n_2/N\). Under \(H_0\), \(\bar X_1-\bar X_2\sim N_p(0,\Sigma/a)\). Consequently, if \(U=\sqrt{a}\,\Sigma^{-1/2}(\bar X_1-\bar X_2)\) and \(W=\nu\Sigma^{-1/2}S_p\Sigma^{-1/2}\), then \(U\sim N_p(0,I_p)\), \(W\sim W_p(I_p,\nu)\), and \(U\) and \(W\) are independent. Substitution into the definition gives \(T^2=\nu U^\top W^{-1}U\).

To obtain the distribution of this quadratic form, write \(R^2=U^\top U\). Then \(R^2\sim\chi^2_p\). Conditional on \(U\), orthogonal invariance of the identity-scale Wishart distribution means that rotating the direction of \(U\) to the first coordinate does not change the distribution of \(W\). It follows that \(U^\top W^{-1}U\) has the same distribution as \(R^2(W^{-1})_{11}\), with the Wishart factor independent of \(R^2\).

For completeness, represent \(W=G^\top G\), where \(G\) is a \(\nu\) by \(p\) matrix of independent standard normal entries. Partition its columns as \(G=[g_1\ G_2]\). The Schur-complement identity gives $$ \frac{1}{(W^{-1})_{11}}=g_1^\top(I-P_{G_2})g_1, $$ where \(P_{G_2}\) is the orthogonal projection onto the column space of \(G_2\). With probability one, that space has dimension \(p-1\) when \(\nu\geq p\). Conditional on \(G_2\), \(g_1\) is an independent standard normal vector, so its squared projection onto the orthogonal complement has a \(\chi^2_{\nu-p+1}\) distribution. This conditional distribution does not depend on \(G_2\); the resulting variable \(V=1/(W^{-1})_{11}\) is therefore independent of \(G_2\), and in particular is independent of \(R^2\). Thus \(U^\top W^{-1}U\) has the distribution of \(R^2/V\), where \(R^2\sim\chi^2_p\) and \(V\sim\chi^2_{\nu-p+1}\) independently. By the definition of an F distribution, $$ \frac{\nu-p+1}{p\nu}T^2 =\frac{(R^2/p)}{(V/(\nu-p+1))} \sim F_{p,\,\nu-p+1}. $$ This proves the result. \(\square\)

The proof also shows where the finite-sample exactness comes from: normal sampling gives independent mean and residual components, while the common covariance permits the two residual sums of squares to be pooled into one Wishart matrix. Without the assumptions that make those steps valid, the displayed F law is not guaranteed.

Worked Applications

Common Pitfalls and Interpretation

A common mistake is to compare \(T^2\) directly with an F critical value. The unscaled Hotelling statistic is not distributed as \(F_{p,\nu-p+1}\); it must first be multiplied by \((\nu-p+1)/(p\nu)\). Another mistake is to use \(\nu\) as the denominator degrees of freedom. The covariance estimate is a \(p\)-dimensional matrix, and the correct denominator degrees of freedom are \(\nu-p+1\).

The condition \(\nu\geq p\) is not a minor technicality. If \(\nu<p\), the pooled residual covariance is singular with probability one under this model, so its inverse and the stated statistic are unavailable. Even when \(\nu\geq p\), a small \(\nu-p+1\) gives a reference distribution with a heavy upper tail; the scaling and degrees of freedom should not be replaced casually by a large-sample approximation.

Finally, exact calibration is a statement about the null distribution under the specified model. It does not establish that the groups actually have a common covariance or that their observations are multivariate normal. If those assumptions are doubtful, the exact F conclusion may not describe the test’s actual behavior. The result concerns the pooled common-covariance test, not tests designed for unequal covariance matrices.