Studentization and the Exact t Distribution

Why Studentize a Contrast?

A contrast estimate has a normal sampling distribution when the errors in a fixed-design linear model are Gaussian. In practice, however, the error variance is usually unknown. Replacing that variance with the residual mean square makes the standard error usable, but it also makes the denominator random. The exact Student t result depends on a particular relationship: the contrast estimate and the residual mean square must be independent, and the residual sum of squares must have a scaled chi-square distribution.

This tutorial derives that relationship and the resulting pivot. Exactness here is a finite-sample property under the stated model, not an approximation that becomes accurate only as the sample size grows.

Model and Definitions

Let $Y$ be an $n$-vector of responses and let $X$ be a fixed $n\times p$ design matrix of rank $p$, with $n>p$. Assume the Gaussian linear model

Y=X\beta+\varepsilon,\qquad \varepsilon\sim N_n(0,\sigma^2 I_n),\qquad \sigma^2>0.

The coefficient vector $\beta$ has $p$ components. A linear contrast is $a^\top\beta$, where $a$ is a specified $p$-vector. It is estimable if $a$ belongs to the row space of $X$; equivalently, there is a vector $d\in\mathbb R^n$ such that $X^\top d=a$. Write $\tau=a^\top\beta$ for the contrast value and $\hat{\tau}=a^\top\hat{\beta}$ for its least-squares estimate. Since $X$ has full column rank, $\hat{\beta}=(X^\top X)^{-1}X^\top Y$.

Definition: For an estimable contrast $a^\top\beta$, define $v_c=a^\top(X^\top X)^{-1}a$. Its estimated variance is $MSE\,v_c$, where $MSE=SSE/\nu$, $SSE=\|Y-X\hat{\beta}\|^2$, and $\nu=n-p$ is the residual degrees of freedom.

If $a\ne 0$ is estimable, then $v_c>0$. Indeed, $X^\top X$ is positive definite because $X$ has full column rank, so its inverse is positive definite and $a^\top(X^\top X)^{-1}a>0$. The restriction $a\ne0$ matters: the zero contrast is estimable, but its variance factor is zero and a standardized ratio for it is undefined.

Why the Contrast and Residuals Are Independent

Let $P_X=X(X^\top X)^{-1}X^\top$, the orthogonal projection onto the column space of $X$. The fitted-value vector is $P_XY$, and the residual vector is $(I_n-P_X)Y=(I_n-P_X)\varepsilon$. For an estimable contrast, its estimation error is a linear function of the error vector:

\hat{\tau}-\tau =a^\top(X^\top X)^{-1}X^\top\varepsilon =d^\top\varepsilon,\qquad d=X(X^\top X)^{-1}a.

The vector $d$ belongs to the column space of $X$, whereas the residual vector lies in its orthogonal complement. This geometric fact yields independence, not merely zero correlation.

Result: Under the fixed-design Gaussian model, the estimation error $\hat{\tau}-\tau$ is independent of the residual vector $(I_n-P_X)Y$, and hence independent of $SSE$.

Proof. Both $d^\top\varepsilon$ and $(I_n-P_X)\varepsilon$ are linear transformations of the same multivariate normal vector, so they are jointly normal. Their cross-covariance is

\operatorname{Cov}\!\left(d^\top\varepsilon,(I_n-P_X)\varepsilon\right) =\sigma^2 d^\top(I_n-P_X)=0,

because $d$ lies in the column space projected onto by $P_X$. Jointly normal random vectors with zero cross-covariance are independent. Since $SSE$ is a function of the residual vector, it too is independent of $\hat{\tau}-\tau$. $\square$

The variance of the contrast estimate follows from the same representation. Since $d=X(X^\top X)^{-1}a$,

\operatorname{Var}(\hat{\tau}-\tau) =\sigma^2 d^\top d =\sigma^2 a^\top(X^\top X)^{-1}a =\sigma^2v_c.

Thus, when $a\ne0$, the standardized error $(\hat{\tau}-\tau)/(\sigma\sqrt{v_c})$ is $N(0,1)$. This normal pivot still uses the unknown $\sigma$; studentization replaces it using the residual mean square.

Residual Degrees of Freedom and the Exact t Pivot

The residual sum of squares has a chi-square distribution after scaling by $\sigma^2$. To see why, choose an orthonormal basis for the $n-p$ dimensional orthogonal complement of the column space of $X$, and put its basis vectors into an $n\times\nu$ matrix $Q$. Then $Q^\top Q=I_\nu$ and $I_n-P_X=QQ^\top$. Consequently,

\frac{SSE}{\sigma^2} =\frac{\varepsilon^\top QQ^\top\varepsilon}{\sigma^2} =\left\|Q^\top\frac{\varepsilon}{\sigma}\right\|^2 \sim\chi^2_\nu.

The last equality in distribution holds because $Q^\top\varepsilon/\sigma\sim N_\nu(0,I_\nu)$, whose squared Euclidean norm is chi-square with $\nu$ degrees of freedom. In particular, $MSE>0$ with probability one when $\nu>0$.

Theorem: Suppose $Y=X\beta+\varepsilon$, with $X$ fixed of rank $p$, $n>p$, and $\varepsilon\sim N_n(0,\sigma^2I_n)$. For every nonzero estimable contrast $a^\top\beta=\tau$, $$ \frac{\hat{\tau}-\tau}{\sqrt{MSE\,v_c}}\sim t_\nu,\qquad v_c=a^\top(X^\top X)^{-1}a,\quad \nu=n-p. $$

Proof. The normal result above gives $Z=(\hat{\tau}-\tau)/(\sigma\sqrt{v_c})\sim N(0,1)$. The residual result gives $U=SSE/\sigma^2\sim\chi^2_\nu$, and the independence proof shows that $Z$ and $U$ are independent. Since $MSE=\sigma^2U/\nu$, direct substitution yields

\frac{\hat{\tau}-\tau}{\sqrt{MSE\,v_c}} = \frac{Z}{\sqrt{U/\nu}}.

By definition, the ratio of an independent standard normal variable to the square root of an independent chi-square variable divided by its degrees of freedom has a Student t distribution with $\nu$ degrees of freedom. Therefore the displayed pivot is $t_\nu$. $\square$

For testing a specified contrast value $\tau_0$, the statistic $(\hat{\tau}-\tau_0)/\sqrt{MSE\,v_c}$ has the $t_\nu$ distribution when the null hypothesis $a^\top\beta=\tau_0$ is true. The residual degrees of freedom come from the dimension left over after fitting the $p$ independent columns of the design, not from the number of coefficients in the contrast.

Worked Applications

Worked Example: An Intercept-Only Model

Suppose $Y_i=\mu+\varepsilon_i$, $i=1,\ldots,5$, with independent errors $N(0,\sigma^2)$. Here $X$ is a column of five ones, $p=1$, and the contrast is $\tau=\mu$, with $a=1$.

The least-squares estimate is $\hat{\mu}=\bar{Y}$, and $X^\top X=5$, so $v_c=1/5$. The residual degrees of freedom are $\nu=5-1=4$. Thus, under the model,

\frac{\bar{Y}-\mu}{\sqrt{MSE/5}}\sim t_4.

For a numerical check, take observed values $10,11,12,13,14$. Their mean is $12$, and the residual sum of squares is $(10-12)^2+(11-12)^2+(12-12)^2+(13-12)^2+(14-12)^2=10$. Hence $MSE=10/4=2.5$. If evaluating the statistic at $\mu=10$, its observed value is $(12-10)/\sqrt{2.5/5}=2/\sqrt{0.5}\approx2.828$. The exact reference distribution under $\mu=10$ is $t_4$.

Worked Example: A Regression Slope

Consider simple linear regression with six fixed predictor values $(-2,-1,0,0,1,2)$. The model is $Y_i=\beta_0+\beta_1x_i+\varepsilon_i$, with independent $N(0,\sigma^2)$ errors. The contrast is the slope $\tau=\beta_1$. Since $\sum_i x_i=0$ and $\sum_i x_i^2=10$, the design columns are orthogonal and $v_c=1/10$. There are $p=2$ fitted coefficients, so $\nu=6-2=4$. The centered slope statistic therefore has distribution

\frac{\hat{\beta}_1-\beta_1}{\sqrt{MSE/10}}\sim t_4.

To check the calculations with actual responses, use $y=(0,-1,4,4,3,8)^\top$. These values equal $3+2x+e$, where $e=(1,-2,1,1,-2,1)^\top$. The residual vector is orthogonal to both design columns: its entries sum to $0$, and $x^\top e=-2+2-2+2=0$. Thus the fitted line has intercept $3$ and slope $2$. The residual sum of squares is $1+4+1+1+4+1=12$, so $MSE=12/4=3$. For testing $\beta_1=0$, the observed statistic is $2/\sqrt{3/10}=2/\sqrt{0.3}\approx3.651$, with $t_4$ as its exact null distribution.

Worked Example: Difference Between Two Group Means

Suppose two independent groups have sizes $3$ and $4$, and observations $4,5,6$ and $6,7,8,9$, respectively. Model the responses with a separate mean for each group and common independent $N(0,\sigma^2)$ errors. The contrast is $\tau=\mu_1-\mu_2$. The fitted means are $\bar{Y}_1=5$ and $\bar{Y}_2=7.5$, so the estimated contrast is $-2.5$.

There are $p=2$ group means and $n=7$ observations, giving $\nu=7-2=5$. The variance factor for the difference is $v_c=1/3+1/4=7/12$. Within-group sums of squares are $2$ and $5$, so $SSE=7$ and $MSE=7/5=1.4$. The estimated standard error is $\sqrt{1.4(7/12)}=\sqrt{0.816\overline{6}}\approx0.904$. Therefore the observed statistic for equality of the means is \[ \frac{(5-7.5)-0}{\sqrt{1.4(7/12)}}\approx-2.766. \] Under the null hypothesis $\mu_1-\mu_2=0$, its exact distribution is $t_5$.

Common Mistakes and Scope

Using the normal pivot after estimating the variance. Replacing $\sigma$ with $\sqrt{MSE}$ changes the distribution. The exact result is t because the standardized contrast error is independent of the chi-square residual sum of squares.
Forgetting the nonzero-contrast condition. If $a=0$, then $\tau=0$, $v_c=0$, and both the standardized numerator and its proposed standard error are zero. The ratio is $0/0$, not a t statistic. The theorem applies only when $a\ne0$ and the contrast is estimable.
Using the wrong degrees of freedom. The denominator uses $\nu=n-p$, the residual degrees of freedom in the fitted model. It does not generally use $n-1$ or a count based only on the contrast.
Assuming the result holds for arbitrary errors. The proof uses Gaussianity for independence and the chi-square law, and spherical covariance $\sigma^2I_n$ for the stated variance and residual distribution. Without these assumptions, the same statistic need not have an exact t distribution.

Studentization is what makes exact small-sample inference possible when the error variance is unknown. Its validity is not guaranteed just because a contrast estimate is approximately normal: the residual variance estimator must have the right scaled chi-square distribution and be independent of the contrast estimate. When those conditions hold, the resulting t reference distribution accounts exactly for the additional uncertainty from estimating $\sigma^2$.

Check Your Understanding

Use the model assumptions and derivation above to answer the following.

Why does estimability place the contrast’s linear estimation error in the column space of $X$?
Which property of the Gaussian model turns zero covariance between the contrast error and residual vector into independence?
For a model with $n=18$ observations and design rank $p=4$, what are the residual degrees of freedom for a nonzero estimable contrast?
What happens to the proposed t ratio if the contrast vector is zero, and why is that case excluded?
In the two-group example, explain why the variance factor is $1/3+1/4$ and not $1/7$.