Exact Joint t Distributions for Correlated Contrasts

Why Correlated Contrasts Have a Joint Distribution

When several contrasts are estimated from the same normal-theory model, their estimates can be correlated: they may reuse the same group means or other fitted quantities. Their standard errors also share the same residual variance estimate. Consequently, a collection of individually valid \(t\) statistics is not generally a collection of independent \(t\) variables. Its joint distribution is what determines exact simultaneous coverage.

We derive that distribution for a full-rank normal linear model. The result identifies two ingredients: the correlation matrix of the contrast-estimation errors and the degrees of freedom of the common variance estimate. It also explains why a single common denominator matters, even when the contrast numerators are uncorrelated.

Definition: Suppose \(Y\sim N(X\beta,\sigma^2 I_n)\), where \(X\) is a fixed \(n\times p\) matrix of rank \(p\), and let \(\hat\beta=(X'X)^{-1}X'Y\). For an \(m\times p\) matrix \(C\), suppose every row contrast in \(C\beta\) is estimable and each diagonal entry of \(V=C(X'X)^{-1}C'\) is positive. Write \(v_{ii}=V_{ii}\), let \(s^2=\mathrm{SSE}/\nu\), and set \(\nu=n-p\). The standardized contrast-error vector is \[ T=D^{-1/2}C(\hat\beta-\beta)/s, \] where \(D=\operatorname{diag}(v_{11},\ldots,v_{mm})\). Its \(i\)th coordinate is the contrast-estimation error divided by its estimated standard error.

For testing specified contrast values, the same vector describes the test statistics under the joint null hypothesis that all those values equal their true contrasts. Define \(R=D^{-1/2}VD^{-1/2}\). This is the correlation matrix of the contrast-estimation errors before they are divided by \(s\). The common denominator does not alter their marginal standardizations, but it is essential to their joint law.

The Exact Joint \(t\) Result

Theorem: Under the normal linear model and the conditions in the definition, \(T\) has the representation \[ T=\frac{Z}{\sqrt{U/\nu}}, \] where \(Z\sim N_m(0,R)\), \(U\sim\chi^2_\nu\), and \(Z\) and \(U\) are independent. If \(R\) is positive definite, \(T\) has a central multivariate \(t\) distribution with \(\nu\) degrees of freedom and scale matrix \(R\). Each coordinate has a univariate \(t_\nu\) distribution. If \(R\) is singular, the same representation holds, but the joint distribution is supported on a lower-dimensional subspace.

The scale matrix is not, in general, the covariance matrix of \(T\). When \(\nu>2\), \[ \operatorname{Cov}(T)=\frac{\nu}{\nu-2}R. \] Thus \(R\) is the correlation matrix of the normal numerators, while the shared denominator inflates the covariance of the \(t\) statistics. The proof establishes both the joint law and this distinction.

Find the numerator distribution.
Since \(\hat\beta-\beta=(X'X)^{-1}X'\varepsilon\) and \(\varepsilon\sim N(0,\sigma^2 I_n)\), the vector \(C(\hat\beta-\beta)/\sigma\) is normal with mean zero and covariance \(V\). Standardizing its coordinates by \(D^{-1/2}\) gives \(Z\sim N_m(0,R)\).

Find the denominator distribution and independence.
The residual sum of squares is \(\mathrm{SSE}=\varepsilon'(I-P_X)\varepsilon\), where \(P_X=X(X'X)^{-1}X'\). The matrices \(P_X\) and \(I-P_X\) project onto orthogonal subspaces. For a normal error vector, the fitted and residual projections are independent. Hence \(Z\) is independent of \(\mathrm{SSE}\), and \(\mathrm{SSE}/\sigma^2\sim\chi^2_\nu\), since \(I-P_X\) has rank \(n-p=\nu\).

Use the common variance estimate.
Writing \(U=\mathrm{SSE}/\sigma^2\) gives \(s^2/\sigma^2=U/\nu\). Dividing each standardized numerator by \(s/\sigma\) yields \(T=Z/\sqrt{U/\nu}\), with the asserted independence.

For positive definite \(R\), the density follows by mixing the conditional normal density of \(T\) given \(U=u\). That density is \[ (2\pi)^{-m/2}|R|^{-1/2}(u/\nu)^{m/2} \exp\{-u\,t'R^{-1}t/(2\nu)\}. \] Multiplying by the chi-square density of \(U\) and integrating over \(u>0\) gives

f_T(t)= \frac{\Gamma((\nu+m)/2)} {\Gamma(\nu/2)(\nu\pi)^{m/2}|R|^{1/2}} \left(1+\frac{t'R^{-1}t}{\nu}\right)^{-(\nu+m)/2}.

Indeed, the integral is a gamma integral with rate \((1+t'R^{-1}t/\nu)/2\); evaluating it gives the displayed normalizing constant. To obtain the covariance, condition on \(U\). Since \(E[Z]=0\), \[ E[T_iT_j]=R_{ij}E[\nu/U] =R_{ij}\frac{\nu}{\nu-2},\qquad \nu>2, \] where \(E[1/U]=1/(\nu-2)\) for \(U\sim\chi^2_\nu\). This proves the covariance formula. \(\square\)

Worked Examples

Worked Example: Two Contrasts of Three Group Means

Consider three independent groups, each with four normal observations and a common variance. Let the group means be \(\mu_1,\mu_2,\mu_3\). The model has \(n=12\), \(p=3\), and \(\nu=9\). Take contrasts \(c_1'\beta=\mu_1-\mu_2\) and \(c_2'\beta=\mu_2-\mu_3\), with \(C=\begin{pmatrix}1&-1&0\\0&1&-1\end{pmatrix}\).

Here \(X'X=4I_3\), so \((X'X)^{-1}=I_3/4\). Therefore \[ V=C(X'X)^{-1}C' =\frac14\begin{pmatrix}2&-1\\-1&2\end{pmatrix} =\begin{pmatrix}1/2&-1/4\\-1/4&1/2\end{pmatrix}. \] Both diagonal entries are \(1/2\), and standardizing gives \(R=\begin{pmatrix}1&-1/2\\-1/2&1\end{pmatrix}\). The two contrast errors are negatively correlated because the second contrast includes \(\hat\mu_2\) with the opposite sign from the first.

Suppose the observed group means are \(10,12,15\), and \(s^2=4\). Each estimated contrast has standard error \(\sqrt{s^2(1/2)}=\sqrt2\). The observed statistics are \(T_1=(-2)/\sqrt2=-\sqrt2\) and \(T_2=(-3)/\sqrt2=-3/\sqrt2\). Under the joint null that both contrasts are zero, their statistics follow the bivariate \(t\) representation with \(\nu=9\) and the displayed \(R\). They are not independent univariate \(t_9\) variables.

Worked Example: Uncorrelated Numerators, Dependent Statistics

Suppose two contrast numerators are uncorrelated, so \(R=I_2\), and use a shared variance estimate with \(\nu=8\). The theorem gives \(T_i=Z_i/\sqrt{U/8}\), where \(Z_1,Z_2\) are independent standard normal variables and both use the same \(U\sim\chi^2_8\).

By symmetry, \(E[T_1T_2]=0\), so the statistics are uncorrelated. But their squares are dependent. Conditional on \(U\), \(T_1^2T_2^2=Z_1^2Z_2^2(8/U)^2\). Independence of the normal numerators and denominator gives \[ E[T_1^2T_2^2]=E[Z_1^2]E[Z_2^2]\,64E[U^{-2}] =64\frac{1}{(8-2)(8-4)}=\frac83. \] Also \(E[T_i^2]=8/(8-2)=4/3\). Consequently, \[ \operatorname{Cov}(T_1^2,T_2^2)=\frac83-\left(\frac43\right)^2=\frac89>0. \] So zero correlation of the numerators does not make the statistics independent: a small shared denominator makes both squared statistics large together.

Worked Example: A Redundant Contrast

Let \(c_2=2c_1\), with \(c_1'\beta\) estimable and positive variance. Then \(\widehat{c_2'\beta}-c_2'\beta=2(\widehat{c_1'\beta}-c_1'\beta)\), and \(v_{22}=4v_{11}\). After standardization by the corresponding standard errors, \(T_2=T_1\) exactly.

The correlation matrix is \(R=\begin{pmatrix}1&1\\1&1\end{pmatrix}\), whose determinant is zero. This is not a failure of the theorem: the joint \(t\) representation still applies, but the distribution lies on the line \(t_2=t_1\), rather than having a two-dimensional density. Requiring both statistics to fall within the same symmetric bound is then equivalent to requiring just one of them to do so.

From the Joint Law to Simultaneous Coverage

Suppose the target contrasts are \(\theta_i=c_i'\beta\), with estimates \(\hat\theta_i=c_i'\hat\beta\). For a common nonnegative critical value \(c\), the simultaneous intervals \[ \hat\theta_i\ \pm\ c\,s\sqrt{v_{ii}},\qquad i=1,\ldots,m, \] cover all their targets exactly when \(\max_i |T_i|\le c\). Thus their familywise coverage probability is determined by the joint \(t\) distribution, not just by the separate marginal \(t_\nu\) distributions.

For a desired coverage level \(1-\alpha\), define a critical value by \[ P_{R,\nu}\!\left(\max_{1\le i\le m}|T_i|\le c\right)\ge 1-\alpha. \] The dependence structure enters through \(R\), and the common variance estimate enters through \(\nu\). Different contrast families can therefore require different exact critical values even if they have the same number of contrasts. This expression states what probability must be calibrated; obtaining max-statistic critical values is a separate calculation.

Common Pitfall

A tempting but incorrect argument is: “Each statistic has a \(t_\nu\) distribution, so the joint probability is the product of the marginal probabilities.” Knowing the marginals does not establish independence. The statistics share \(s\), and even when their normal numerators are independent, the example above shows that their squares are dependent. The correction is to use the joint representation \(Z/\sqrt{U/\nu}\), which retains both the numerator correlation matrix and the common denominator.

A second point worth keeping distinct is that \(R\) is a scale and numerator-correlation matrix, not generally the covariance matrix of \(T\). For \(\nu>2\), the latter is \(\nu R/(\nu-2)\). For simultaneous coverage, use the joint distribution itself rather than treating \(R\) as a covariance matrix for the final statistics.

Check Your Understanding

Use the joint representation to answer these questions.

In the normal linear model, what are the distributions of \(Z\) and \(U\), and why are they independent?
For two contrasts with independent normal numerators and a shared variance estimate, explain why the resulting \(t\) statistics need not be independent.
When \(\nu>2\), how does \(\operatorname{Cov}(T)\) relate to the scale matrix \(R\)?
What does a singular contrast correlation matrix imply about the joint distribution?
Why is the probability of simultaneous coverage not generally the product of the marginal coverage probabilities?