Wald Test From the Unrestricted Estimate

Why the Unrestricted Estimate Gives a Wald Test

A Wald test asks whether the unrestricted estimate lies far from the set of coefficient values allowed by a null hypothesis, after measuring that distance in units of its sampling uncertainty. For a multivariate response, the restriction and its estimate are matrices. The Wald statistic combines their entries while accounting for dependence both among coefficients and among responses.

The key construction is to estimate the restriction in the unrestricted model, find its covariance, and form a squared distance using the inverse covariance. This makes the test sensitive to the uncertainty and correlation structure rather than simply summing unscaled squared discrepancies.

Model, Restriction, and Covariance

Consider the Gaussian multivariate linear model

where \(Y\) is an \(n\times p\) response matrix, \(X\) is a fixed \(n\times k\) design matrix of full column rank, and \(B\) is a \(k\times p\) coefficient matrix. The rows of \(U\) are independent \(N_p(0,\Sigma)\), with \(\Sigma\) positive definite. Write \(G=(X'X)^{-1}\) and let \(\hat B=GY'_{\!X}\), equivalently \(\hat B=(X'X)^{-1}X'Y\), be the unrestricted least-squares estimate.

We test the matrix-valued linear restriction \(CBM=\Theta\). Here \(C\) is a fixed \(q\times k\) matrix of row rank \(q\), \(M\) is a fixed \(p\times s\) matrix of column rank \(s\), and \(\Theta\) is a specified \(q\times s\) matrix. Thus \(CBM\) selects and combines coefficient directions and response directions. Its unrestricted estimate is \(C\hat B M\), and its discrepancy from the null value is

For a matrix \(A\), \(\operatorname{vec}(A)\) stacks its columns into a vector. We use the identity \(\operatorname{vec}(ABC)=(C'\otimes A)\operatorname{vec}(B)\), where \(\otimes\) denotes the Kronecker product.

Deriving the Wald Statistic

Since \(\operatorname{vec}(Y)=\operatorname{vec}(XB)+\operatorname{vec}(U)\), the covariance of \(\operatorname{vec}(U)\) is \(\Sigma\otimes I_n\). The least-squares estimate satisfies \(\hat B-B=GX'U\). Applying the vec identity gives

The covariance rule \(\operatorname{Cov}(AZ)=A\operatorname{Cov}(Z)A'\) therefore yields

Using the vec identity again for \(C(\hat B-B)M\), its vectorized form is \((M'\otimes C)\operatorname{vec}(\hat B-B)\). Consequently,

Both factors are positive definite: \(CGC'\) is positive definite because \(G\) is positive definite and \(C\) has full row rank; \(M'\Sigma M\) is positive definite because \(\Sigma\) is positive definite and \(M\) has full column rank. The covariance is therefore invertible, with inverse \((M'\Sigma M)^{-1}\otimes(CGC')^{-1}\).

has a \(\chi^2_{qs}\) distribution.

Proof. Under the null, \(E[\operatorname{vec}(C\hat B M)]=\operatorname{vec}(\Theta)\). The estimate is a linear transformation of the Gaussian matrix \(Y\), so the centered vector \(\operatorname{vec}(D)\) is multivariate normal with mean zero and the covariance just derived. For any mean-zero normal vector \(Z\) with positive-definite covariance \(\Omega\), \(\Omega^{-1/2}Z\) is normal with identity covariance. Its squared Euclidean length is chi-square with degrees of freedom equal to its dimension. Here that dimension is \(qs\), so \(W_0\sim\chi^2_{qs}\). The trace expression follows from the vec identity for quadratic forms and the Kronecker inverse rule. \(\square\)

The degrees of freedom count the \(qs\) independent entries of the restricted matrix, not the total number \(kp\) of coefficients. The rank conditions above ensure that the restriction covariance has that full dimension.

Unknown Error Covariance

In practice, \(\Sigma\) is usually unknown. Let \(P_X=X(X'X)^{-1}X'\), let \(\nu=n-k\), and define the unrestricted residual sum-of-squares-and-cross-products matrix \(E=Y'(I_n-P_X)Y\). When \(\nu>0\), the usual unbiased estimator is \(\hat\Sigma=E/\nu\). Replacing \(\Sigma\) by \(\hat\Sigma\) gives the plug-in Wald statistic

This statistic is defined when \(M'\hat\Sigma M\) is invertible. Under the Gaussian model, \(E\) has a Wishart distribution with \(\nu\) degrees of freedom; in particular, it is positive definite with probability one when \(\nu\ge p\). Since \(M\) has full column rank, this condition ensures that \(M'\hat\Sigma M\) is positive definite with probability one.

The plug-in statistic is not, in general, exactly \(\chi^2_{qs}\) in a finite sample. Its familiar reference is asymptotic. For example, suppose \(k,p,q,s,C,M\) are fixed, \(\nu\) tends to infinity, and the design satisfies the usual information-growth condition \(X'X/n\to Q\) for a positive-definite matrix \(Q\). Under the null, \(\hat\Sigma\xrightarrow{p}\Sigma\), so the estimated covariance of the restricted estimate is consistent for its true covariance. The standardized restricted estimate is asymptotically standard normal by the Gaussian model and this covariance consistency; the continuous mapping theorem then gives \(W\xrightarrow{d}\chi^2_{qs}\). This is the basis for using the chi-square tail probability as an approximate p-value when \(\Sigma\) is estimated.

Worked Wald Calculations

Common Pitfall and Interpretation

A common mistake is to calculate \(\sum_{i,j}D_{ij}^2\) and treat it as the Wald statistic. That sum ignores the scale and correlations in the estimate. For instance, a discrepancy of one unit may be substantial if its standard error is small and unremarkable if its standard error is large. The correction is to use the inverse covariance of the entire vectorized restriction, or equivalently the two inverse covariance factors in the trace formula.

The Wald statistic measures distance from the null restriction in the uncertainty geometry of the unrestricted estimate. With known \(\Sigma\), its null calibration is exact under the stated Gaussian model; with estimated \(\Sigma\), the chi-square calibration is an asymptotic approximation, not an automatic finite-sample identity. The statistic’s construction also makes clear why the rank and invertibility conditions matter: a redundant restriction or singular covariance cannot be handled by the displayed inverse without changing the formulation.