Sampling Distribution of an Estimable Contrast

Why the Sampling Distribution of a Contrast Matters

Least-squares analysis often targets a combination of coefficients rather than a single coefficient. A scientific question might concern a difference between two treatment means, a slope, or the predicted mean response at a specified covariate value. Each is a linear contrast of the model coefficients. Its sampling distribution describes how its least-squares estimate would vary across repeated responses under the assumed model.

Here the goal is to derive that distribution and express its variance in terms of the design matrix. The result is exact under a fixed design and Gaussian errors. It is a normal-distribution result; by itself, it does not provide the studentized distribution or a test statistic.

Definition: In the linear model \(Y=X\beta+\varepsilon\), a linear contrast is a specified linear function \(c'\beta\), where \(c\) is a fixed vector. A contrast is estimable if its value is determined by the model’s mean vector \(X\beta\). Equivalently, \(c\) belongs to the row space of \(X\). An estimator is a statistic used to estimate a parameter; it need not be unbiased. An estimator is unbiased when its expectation equals the parameter for every allowed value of that parameter.

If \(X\) has full column rank, every linear function of \(\beta\) is estimable. If \(X\) is rank deficient, some coefficient combinations cannot be determined separately, but particular contrasts may still be estimable. The distinction matters: a least-squares coefficient vector may not be unique, while an estimable contrast has the same value for every coefficient vector that gives the fitted mean.

Model and Main Distribution Result

Treat the design matrix \(X\), of size \(n\times p\), as fixed. Assume \(Y=X\beta+\varepsilon\), where \(\varepsilon\sim N_n(0,\sigma^2 I_n)\) and \(\sigma^2>0\). Write \(r=\operatorname{rank}(X)\). To describe least squares even if \(X\) is rank deficient, let \(B\) be a \(p\times r\) matrix whose columns form an orthonormal basis for the row space of \(X\). Then \(B'B=I_r\), and \(Z=XB\) has full column rank \(r\).

Every estimable contrast can be written \(c=B a\) for a unique \(r\)-vector \(a\). The model mean is \(X\beta=Z\gamma\), where \(\gamma=B'\beta\). Thus the fitted model can be expressed using the identifiable parameter \(\gamma\). Least squares in this reduced model gives \(\hat{\gamma}=(Z'Z)^{-1}Z'Y\). The corresponding estimate of the contrast is \(a'\hat{\gamma}\), or equivalently \(c'\hat{\beta}\) for any least-squares coefficient vector \(\hat{\beta}\).

Theorem: Under the fixed-design Gaussian linear model above, if \(c'\beta\) is estimable, its least-squares estimator is unbiased and has sampling distribution

c'\hat{\beta}\sim N\!\left(c'\beta,\ \sigma^2 c'(X'X)^+c\right),

where \((X'X)^+\) is the Moore–Penrose inverse. If \(X\) has full column rank, this reduces to the familiar expression with \((X'X)^{-1}\) in place of \((X'X)^+\).

Proof. Since \(Z\) has full column rank, its least-squares estimator is \(\hat{\gamma}=(Z'Z)^{-1}Z'Y\). Substitute \(Y=Z\gamma+\varepsilon\):

\hat{\gamma} =\gamma+(Z'Z)^{-1}Z'\varepsilon.

The error vector is multivariate normal, so this affine transformation is multivariate normal. Its expectation is \(\gamma\), because \(E[\varepsilon]=0\), and its covariance matrix is

\begin{aligned} \operatorname{Cov}(\hat{\gamma}) &=(Z'Z)^{-1}Z'\operatorname{Cov}(\varepsilon)Z(Z'Z)^{-1}\\ &=\sigma^2(Z'Z)^{-1}Z'Z(Z'Z)^{-1}\\ &=\sigma^2(Z'Z)^{-1}. \end{aligned}

Therefore \(a'\hat{\gamma}\) is normal, with expectation \(a'\gamma=c'\beta\) and variance \(\sigma^2a'(Z'Z)^{-1}a\). To express this variance using \(X\), note that \(X=ZB'\): every row of \(X\) lies in the row space spanned by \(B'\). Hence \(X'X=B(Z'Z)B'\), and its Moore–Penrose inverse is \(B(Z'Z)^{-1}B'\). Since \(c=Ba\),

c'(X'X)^+c =a'B'B(Z'Z)^{-1}B'Ba =a'(Z'Z)^{-1}a.

This proves the stated distribution and variance formula. In particular, the contrast estimator is unbiased; that property follows from the model and estimability, not from the word “estimator” alone. \(\square\)

Full-Rank Formula and Interpretation

When \(X\) has full column rank, the least-squares estimator is \(\hat{\beta}=(X'X)^{-1}X'Y\). Substitution into the model gives \(\hat{\beta}=\beta+(X'X)^{-1}X'\varepsilon\). Thus the coefficient vector is multivariate normal, with mean \(\beta\) and covariance matrix \(\sigma^2(X'X)^{-1}\). Taking a fixed linear combination yields the following formula.

Result: For a full-column-rank design and any fixed vector \(c\),

c'\hat{\beta}\sim N\!\left(c'\beta,\ \sigma^2c'(X'X)^{-1}c\right).

The variance depends on both the contrast and the design. A contrast can have high variance when its coefficients combine directions that the observed design estimates imprecisely. The expression \(c'(X'X)^{-1}c\) is nonnegative, since \((X'X)^{-1}\) is positive definite. If \(c\ne 0\), it is positive; multiplying by \(\sigma^2>0\) gives a positive sampling variance.

The distribution is conditional on the fixed design and the model parameters. If \(\sigma^2\) is unknown, the normal distribution still describes the sampling distribution when the true \(\sigma^2\) is held fixed. Replacing \(\sigma^2\) by a residual estimate changes the statistic and requires a separate distributional argument.

Worked Example: A Slope and a Fitted Mean

Suppose \(Y_i=\beta_0+\beta_1x_i+\varepsilon_i\), with fixed predictor values \(x=0,1,2\) and independent errors \(N(0,\sigma^2)\). The observed responses are \(3,5,8\). Find the least-squares slope and its sampling distribution, then find the estimated mean at \(x=2\) and its sampling distribution.

The design matrix and cross-product are

X= \begin{pmatrix}1&0\\1&1\\1&2\end{pmatrix}, \qquad X'X= \begin{pmatrix}3&3\\3&5\end{pmatrix}, \qquad (X'X)^{-1} =\frac{1}{6} \begin{pmatrix}5&-3\\-3&3\end{pmatrix}.

The response mean is \(16/3\), the predictor mean is \(1\), and \(\sum_i(x_i-\bar{x})^2=2\). The slope estimate is \[ \hat{\beta}_1 =\frac{\sum_i(x_i-\bar{x})(Y_i-\bar{Y})}{\sum_i(x_i-\bar{x})^2} =\frac{(-1)(-7/3)+(1)(8/3)}{2} =\frac{5}{2}. \]

For the slope, \(c=(0,1)'\). The second diagonal entry of \((X'X)^{-1}\) is \(3/6=1/2\), so \(\hat{\beta}_1\sim N(\beta_1,\sigma^2/2)\). The least-squares intercept is \(\hat{\beta}_0=\bar{Y}-\hat{\beta}_1\bar{x}=16/3-5/2=17/6\).

The mean response at \(x=2\) is \(\beta_0+2\beta_1\), so use \(c=(1,2)'\). Its estimate is \(\hat{\beta}_0+2\hat{\beta}_1=17/6+5=47/6\). Direct multiplication gives \[ (X'X)^{-1}c =\begin{pmatrix}-1/6\\1/2\end{pmatrix}, \qquad c'(X'X)^{-1}c=-1/6+1=5/6. \] Consequently, the estimated mean at \(x=2\) has distribution \(N(\beta_0+2\beta_1,5\sigma^2/6)\). The contrast and its variance are determined by the design, not by the particular observed responses.

Worked Example: Difference Between Two Group Means

Consider two independent groups under a common-variance normal model. Each observation in group 1 has mean \(\mu_1\), and each in group 2 has mean \(\mu_2\). There are two observations in group 1, \(4\) and \(6\), and three in group 2, \(2,3,7\). Find the least-squares estimate and sampling distribution of \(\mu_1-\mu_2\).

With indicator columns for the two groups, \(X'X=\operatorname{diag}(2,3)\). The coefficient vector is \(\beta=(\mu_1,\mu_2)'\), and the contrast vector is \(c=(1,-1)'\). The group means are \(\hat{\mu}_1=(4+6)/2=5\) and \(\hat{\mu}_2=(2+3+7)/3=4\). Therefore the contrast estimate is \(c'\hat{\beta}=5-4=1\).

The inverse cross-product is \(\operatorname{diag}(1/2,1/3)\), so \[ c'(X'X)^{-1}c =(1,-1) \begin{pmatrix}1/2&0\\0&1/3\end{pmatrix} \begin{pmatrix}1\\-1\end{pmatrix} =\frac12+\frac13=\frac56. \] Thus \(\hat{\mu}_1-\hat{\mu}_2\sim N(\mu_1-\mu_2,5\sigma^2/6)\). This is a distributional statement under the model; it does not assert that the observed difference \(1\) equals the true difference.

Estimable Contrasts When the Design Is Rank Deficient

A rank-deficient design does not make every contrast unusable. It means the data do not identify every coordinate of \(\beta\) separately. A contrast is estimable precisely when it is unchanged by moving \(\beta\) in any direction \(v\) for which \(Xv=0\). Indeed, \(c'\beta\) is unchanged under \(\beta\mapsto\beta+v\) exactly when \(c'v=0\) for every \(v\) in the null space of \(X\). The orthogonal complement of that null space is the row space of \(X\), giving the stated condition \(c\in\operatorname{Row}(X)\).

Worked Example: An Estimable Contrast Despite Redundant Coefficients

Let the design rows be \((1,0,1)\), \((1,1,1)\), and \((1,2,1)\), with responses \(2,4,6\). The model mean at predictor value \(x\) is \(\beta_0+\beta_1x+\beta_2\). The first and third columns are identical, so \(\beta_0\) and \(\beta_2\) cannot be identified separately. Determine whether \(\beta_0+\beta_2\) is estimable and find the variance of its least-squares estimator.

Write \(\theta=\beta_0+\beta_2\) and \(\delta=\beta_1\). The identifiable model is \(Y_i=\theta+\delta x_i+\varepsilon_i\), with reduced design \[ Z=\begin{pmatrix}1&0\\1&1\\1&2\end{pmatrix}, \qquad Z'Z=\begin{pmatrix}3&3\\3&5\end{pmatrix}. \] The target \(\beta_0+\beta_2=\theta\) is determined by this reduced model, so it is estimable. The observed data lie exactly on the line with intercept \(2\) and slope \(2\), giving \(\hat{\theta}=2\) and \(\hat{\delta}=2\).

The contrast vector in the original parameterization is \(c=(1,0,1)'\), which lies in the row space of \(X\). Its estimator is \(\hat{\theta}\), regardless of how a least-squares solution allocates the fitted intercept between \(\hat{\beta}_0\) and \(\hat{\beta}_2\). Since \[ (Z'Z)^{-1} =\frac16\begin{pmatrix}5&-3\\-3&3\end{pmatrix}, \] the variance of \(\hat{\theta}\), the first reduced-model coefficient, is \(5\sigma^2/6\). Hence \(\hat{\beta}_0+\hat{\beta}_2\sim N(\beta_0+\beta_2,5\sigma^2/6)\). The observed residuals happen to be zero, but that does not imply the model variance \(\sigma^2\) is zero.

Common Pitfall and Practical Meaning

Confusing “estimator” with “unbiased estimator.” An estimator is not unbiased by definition. Here unbiasedness is a consequence of the linear model and the contrast’s estimability.
Using a coefficient covariance formula without checking rank. The inverse \((X'X)^{-1}\) requires full column rank. For a rank-deficient design, use an estimable contrast and the generalized-inverse formula, or work in identifiable reduced coordinates.
Dropping covariance terms in a contrast variance. In general, \(\operatorname{Var}(c'\hat{\beta})\) depends on the full covariance matrix, including off-diagonal entries. The compact expression \(c'(X'X)^{-1}c\,\sigma^2\) includes those terms.
Calling the result an exact t distribution. The result proved here is normal when \(\sigma^2\) is fixed. Studentization introduces a residual variance estimate and needs its own argument.

The contrast distribution is useful because it translates a design and a scientific target into an exact sampling variance. The same calculation applies to slopes, differences of means, and fitted means, provided the target is estimable and the stated Gaussian model holds. The design controls precision through \(X'X\); the observed response determines the contrast estimate, but not its model-based variance factor.

Check Your Understanding

Use the fixed-design Gaussian model unless a question states otherwise.

In a full-column-rank model, what are the mean and variance of \(c'\hat{\beta}\)?
Why does a rank-deficient design require an estimability condition before interpreting a coefficient contrast?
For a two-group difference with group sizes \(4\) and \(6\), what is the variance of the estimated difference under a common error variance \(\sigma^2\)?
Explain why an estimator need not be unbiased, and identify why the contrast estimator in this tutorial is unbiased.
Why does the normal sampling distribution derived here not, by itself, establish an exact t distribution?