Computing Exact Hotelling P-Values

From an Observed Statistic to a P-Value

A one-sample Hotelling test compares a vector of sample means with a specified null mean while estimating the covariance among the measured variables. Once the test statistic has been calculated, inference requires a calibration: which reference distribution should it be compared with, and which tail represents evidence against the null? Under the multivariate normal model, the answer is an exact F distribution, not an asymptotic approximation.

This tutorial focuses on carrying out that calibration and using it to calculate a p-value or a critical-value decision. The degrees of freedom matter: using the wrong factor or reversing the tail can change the conclusion.

Definition: Let $X_1,\ldots,X_n$ be independent observations from $N_p(\mu,\Sigma)$, where $\Sigma$ is positive definite, and consider $H_0:\mu=\mu_0$. Write $\bar X=n^{-1}\sum_{i=1}^n X_i$ and $S=(n-1)^{-1}\sum_{i=1}^n(X_i-\bar X)(X_i-\bar X)'$. When $n>p$, the one-sample Hotelling statistic is $$ T^2=n(\bar X-\mu_0)'S^{-1}(\bar X-\mu_0). $$ Under $H_0$, its exact calibration is $$ F_{\mathrm{obs}}=\frac{n-p}{p(n-1)}T^2\ \sim\ F_{p,n-p}. $$

Here $p$ is the dimension of the mean vector, and $F_{p,n-p}$ denotes an F distribution with numerator degrees of freedom $p$ and denominator degrees of freedom $n-p$. The calibration assumes independent multivariate normal observations and a nonsingular sample covariance matrix. Under these assumptions with $n>p$, $S$ is nonsingular with probability one. The null hypothesis is rejected in the direction of large $T^2$, so the relevant probability is an upper-tail probability.

Computing the Exact Upper-Tail Probability

Let $G$ be the cumulative distribution function of $F_{p,n-p}$. The exact p-value for an observed statistic $T^2_{\mathrm{obs}}$ is the probability, under $H_0$, of obtaining a calibrated statistic at least as large as the one observed:

p_{\mathrm{val}} =P_{H_0}\!\left(F_{p,n-p}\geq \frac{n-p}{p(n-1)}T^2_{\mathrm{obs}}\right) =1-G(F_{\mathrm{obs}}).

Because the F distribution is continuous, using “greater than” or “greater than or equal to” gives the same probability. Some F tables report upper-tail probabilities directly; others report quantiles. When a table is unavailable, the upper tail can be expressed using the regularized incomplete beta function:

Result: For $x>0$, define $I_z(a,b)=B(a,b)^{-1}\int_0^z t^{a-1}(1-t)^{b-1}\,dt$, where $B(a,b)=\int_0^1 t^{a-1}(1-t)^{b-1}\,dt$. Then $$ P(F_{p,\nu}\geq x) =I_{\nu/(\nu+px)}\!\left(\frac{\nu}{2},\frac{p}{2}\right), \qquad \nu>0. $$

To see the transformation, the density of $F_{p,\nu}$ is $$ f(x)=\frac{(p/\nu)^{p/2}}{B(p/2,\nu/2)} x^{p/2-1}\left(1+\frac{px}{\nu}\right)^{-(p+\nu)/2},\qquad x>0. $$ In its upper-tail integral from $x$ to infinity, set $u=\nu/(\nu+py)$, where $y$ is the integration variable. The endpoint $y=x$ maps to $u=\nu/(\nu+px)$, and $y\to\infty$ maps to $u\to0$. Substitution, including the Jacobian, gives the integral of $u^{\nu/2-1}(1-u)^{p/2-1}/B(\nu/2,p/2)$ from zero to $\nu/(\nu+px)$. This is the stated $I_z(\nu/2,p/2)$.

A convenient simplification occurs when $p=2$. Since $I_z(\nu/2,1)=z^{\nu/2}$, $$ P(F_{2,\nu}\geq x)=\left(\frac{\nu}{\nu+2x}\right)^{\nu/2}. $$ The simplification is useful for hand calculations, but the general beta-function formula applies for any positive numerator and denominator degrees of freedom.

Worked Calculations

Worked Example: A Two-Variable Test With Six Observations

Suppose $p=2$, $n=6$, the observed sample covariance is $S=I_2$, and $\bar X-\mu_0=(1,1)'$. Calculate the exact p-value and make a test decision at level $\alpha=0.10$.

First compute the quadratic form. Since $S^{-1}=I_2$, $$ T^2_{\mathrm{obs}} =6(1,1)I_2(1,1)'=6(1^2+1^2)=12. $$ The calibration factor is $(n-p)/(p(n-1))=4/(2\cdot5)=0.4$, so $F_{\mathrm{obs}}=0.4(12)=4.8$, with degrees of freedom $(2,4)$. The $p=2$ upper-tail formula gives $$ p_{\mathrm{val}} =\left(\frac{4}{4+2(4.8)}\right)^2 =\left(\frac{4}{13.6}\right)^2 =\left(\frac{5}{17}\right)^2 =\frac{25}{289}\approx0.0865. $$ Since $0.0865<0.10$, reject $H_0$ at the 10% level.

The critical-value calculation agrees. Solving $P(F_{2,4}\geq x)=0.10$ gives $x=2(\sqrt{10}-1)\approx4.3249$. On the $T^2$ scale this is $2.5x\approx10.812$. The observed $T^2=12$ exceeds that cutoff.

Worked Example: A Two-Variable Test Near the 5% Boundary

Now let $p=2$, $n=10$, $S=I_2$, and $\bar X-\mu_0=(9/(4\sqrt{5}),0)'$. Test at level $\alpha=0.05$.

The squared length of the difference vector is $81/80$, so $$ T^2_{\mathrm{obs}}=10\left(\frac{81}{80}\right)=\frac{81}{8}=10.125. $$ Here the calibration factor is $8/(2\cdot9)=4/9$. Therefore $F_{\mathrm{obs}}=(4/9)(81/8)=9/2$, with degrees of freedom $(2,8)$. Its exact upper-tail probability is $$ p_{\mathrm{val}} =\left(\frac{8}{8+2(9/2)}\right)^4 =\left(\frac{8}{17}\right)^4 =\frac{4096}{83521}\approx0.04904. $$ This is just below $0.05$, so the test rejects at the 5% level.

For comparison, the 5% critical F value solves $(8/(8+2x))^4=0.05$, giving $x=4(20^{1/4}-1)\approx4.4590$. The corresponding $T^2$ cutoff is $(9/4)x\approx10.033$. Since $10.125>10.033$, the critical-value rule also rejects.

Worked Example: Three Variables and a Symmetric F Tail

Suppose $p=3$, $n=6$, $S=I_3$, and $\bar X-\mu_0=(\sqrt{5/6},0,0)'$. Find the exact p-value and decide at level $\alpha=0.05$.

The squared length of the difference vector is $5/6$, so $T^2_{\mathrm{obs}}=6(5/6)=5$. The calibration factor is $(6-3)/(3(6-1))=1/5$, hence $F_{\mathrm{obs}}=1$, with degrees of freedom $(3,3)$. The beta-function argument is $$ \frac{\nu}{\nu+px}=\frac{3}{3+3(1)}=\frac12. $$ Thus $p_{\mathrm{val}}=I_{1/2}(3/2,3/2)=1/2$. The last equality follows because the beta density with equal parameters is symmetric around $1/2$, so half its probability lies on either side. Since $0.5>0.05$, do not reject $H_0$.

Why the P-Value and Critical-Value Rules Agree

A critical-value test and a p-value test are two ways to express the same upper-tail comparison. Let $F_{p,n-p;1-\alpha}$ denote the $1-\alpha$ quantile of the F distribution, so that its upper tail above this quantile has probability $\alpha$. The corresponding cutoff on the $T^2$ scale is obtained by reversing the calibration:

Result: At level $\alpha$, reject $H_0$ if and only if $$ F_{\mathrm{obs}}\geq F_{p,n-p;1-\alpha}, \qquad\text{equivalently}\qquad T^2_{\mathrm{obs}}\geq \frac{p(n-1)}{n-p}F_{p,n-p;1-\alpha}. $$

For the equivalence, the calibration multiplier $(n-p)/(p(n-1))$ is positive because $n>p$. Multiplying the F-scale inequality by its reciprocal therefore preserves the inequality and gives the stated $T^2$ cutoff. By the definition of the quantile and the upper-tail p-value, $$ F_{\mathrm{obs}}\geq F_{p,n-p;1-\alpha} \quad\Longleftrightarrow\quad 1-G(F_{\mathrm{obs}})\leq\alpha \quad\Longleftrightarrow\quad p_{\mathrm{val}}\leq\alpha. $$ The F distribution is continuous, so equality at the cutoff has probability zero under the null; the choice of strict or non-strict inequality does not affect the test’s size. $\square$

What “Exact” Means—and a Common Pitfall

Under the null and the stated normal-model assumptions, the calibrated statistic has the specified F distribution exactly in finite samples. This also means that its upper-tail p-value is uniform under $H_0$: if $G$ is the continuous, strictly increasing F cumulative distribution function, then $U=1-G(F_{\mathrm{obs}})$ satisfies $P_{H_0}(U\leq u)=u$ for $0<u<1$. Indeed, the event $U\leq u$ is the event $F_{\mathrm{obs}}\geq G^{-1}(1-u)$, whose probability is $u$ by the definition of the quantile. In particular, rejecting when $p_{\mathrm{val}}\leq\alpha$ has null rejection probability $\alpha$.

“Exact” refers to the reference distribution under the model, not to certainty about the scientific conclusion. A p-value is not the probability that $H_0$ is true; it is a tail probability calculated assuming $H_0$ and the model conditions. If the multivariate normal assumption is not suitable, the exact finite-sample calibration need not apply.

Wrong: Compare $T^2_{\mathrm{obs}}$ directly with an F critical value. Correction: First convert $T^2$ using $(n-p)/(p(n-1))$, or convert the F critical value back to the $T^2$ scale.
Wrong: Use a lower-tail probability because the test is written in terms of a distribution function. Correction: Large deviations from the null make $T^2$ large, so calculate $1-G(F_{\mathrm{obs}})$.
Wrong: Treat a small p-value as the probability that the null is true. Correction: It is the probability, assuming the null and model, of a statistic at least this large.

Key takeaway: For a one-sample Hotelling test, scale the observed statistic to $F_{\mathrm{obs}}=(n-p)T^2_{\mathrm{obs}}/[p(n-1)]$, use the upper tail of $F_{p,n-p}$, and reject at level $\alpha$ exactly when the p-value is at most $\alpha$.

Check Your Understanding

For each question, keep the degrees of freedom and the direction of the tail explicit.

For $p=4$ and $n=12$, what are the numerator and denominator degrees of freedom for the exact F calibration?
If $T^2_{\mathrm{obs}}=18$, $p=3$, and $n=9$, calculate $F_{\mathrm{obs}}$.
Why is the exact Hotelling p-value an upper-tail probability rather than a lower-tail probability?
State the relationship between the p-value decision rule and the F critical-value rule at level $\alpha$.
In what sense is the p-value “exact,” and which model assumption is essential to the finite-sample calibration?