Wald Tests From Gaussian Likelihood Estimates

Why Wald Tests Use the Unrestricted Estimate

A Wald test asks how far the unrestricted likelihood estimate lies from the null restriction, measured in units of its estimated uncertainty. For a Gaussian model, the parameter may contain a mean vector, a covariance matrix, or both. A linear restriction leads directly to a quadratic form. A smooth nonlinear restriction instead requires a local linear approximation, so its gradient determines how parameter uncertainty translates into uncertainty in the restriction.

The construction uses the unrestricted estimator, not an estimate fitted under the null. This makes the test convenient to calculate from a likelihood estimate and its covariance, but the result is a local, covariance-scaled measure of departure from the null. Its chi-square calibration requires regularity conditions.

Definition: Let $\theta\in\mathbb{R}^d$ be an interior parameter in a regular model, and let $\hat{\theta}$ be its unrestricted maximum likelihood estimator. Suppose the null hypothesis is $H_0:g(\theta)=0$, where $g:\mathbb{R}^d\to\mathbb{R}^q$ is continuously differentiable near the null value. Write $G(\theta)=\partial g(\theta)/\partial\theta^\mathsf{T}$ for its $q$ by $d$ Jacobian, and let $\widehat V$ estimate the covariance matrix of $\hat{\theta}$. The Wald statistic is $$ W=\ g(\hat{\theta})^\mathsf{T} \left\{G(\hat{\theta})\widehat V G(\hat{\theta})^\mathsf{T}\right\}^{-1} g(\hat{\theta}), $$ provided the matrix being inverted is nonsingular. The test rejects for large values of $W$.

The covariance inside the quadratic form is the estimated covariance of the restriction vector, obtained by linearizing $g$ at $\hat{\theta}$. If the model is based on $n$ independent observations and $\widehat V$ estimates the covariance of $\hat{\theta}$, it is typically of order $1/n$. In a regular likelihood model, an appropriate estimate may be obtained from the inverse information matrix, when that estimate is justified. The key point here is that $\widehat V$ must be on the covariance scale of $\hat{\theta}$, not the scale of a single observation.

Affine Restrictions on a Gaussian Mean

Suppose $X_1,\ldots,X_n$ are independent $p$-dimensional observations from $N_p(\mu,\Sigma)$, with known positive-definite $\Sigma$. The unrestricted Gaussian likelihood estimate is $\hat{\mu}=\bar X$, and its covariance is $\Sigma/n$. For a rank-$q$ matrix $A$, consider the affine null hypothesis $H_0:A\mu=b$. Take $g(\mu)=A\mu-b$, whose Jacobian is $A$. Substitution gives

W=(A\bar X-b)^\mathsf{T} \left(A\frac{\Sigma}{n}A^\mathsf{T}\right)^{-1} (A\bar X-b) =n(A\bar X-b)^\mathsf{T}(A\Sigma A^\mathsf{T})^{-1}(A\bar X-b).

The degrees of freedom are the number $q$ of independent restrictions, not the dimension $p$ of the mean vector. When the Gaussian model and covariance are known as above, this particular statistic has an exact chi-square distribution with $q$ degrees of freedom under the null. In more general regular likelihood settings, the corresponding chi-square statement is asymptotic.

Worked Example: A Two-Component Gaussian Mean

Suppose four independent observations are $X_1=(0,0)^\mathsf{T}$, $X_2=(2,0)^\mathsf{T}$, $X_3=(0,1)^\mathsf{T}$, and $X_4=(2,1)^\mathsf{T}$. Their mean is $\bar X=(1,0.5)^\mathsf{T}$. Assume $X_i\sim N_2(\mu,\Sigma)$ with known $\Sigma=\begin{pmatrix}2&1\\1&2\end{pmatrix}$, and test $H_0:\mu=(0,0)^\mathsf{T}$.

Here $A$ is the two-dimensional identity matrix, $b=(0,0)^\mathsf{T}$, and there are $q=2$ restrictions. The inverse covariance is $$ \Sigma^{-1}=\frac{1}{3}\begin{pmatrix}2&-1\\-1&2\end{pmatrix}. $$

The observed quadratic form is $$ W=4\bar X^\mathsf{T}\Sigma^{-1}\bar X =\frac{4}{3}\left\{2(1)^2-2(1)(0.5)+2(0.5)^2\right\} =\frac{4}{3}(1.5)=2. $$

Under the null, $W\sim\chi^2_2$ exactly in this known-covariance Gaussian model. The calculation uses the covariance $\Sigma/4$ of the sample mean and tests both mean components jointly.

Nonlinear Restrictions and Local Linearization

For nonlinear $g$, the Wald statistic still compares the estimated restriction $g(\hat{\theta})$ with its estimated covariance, but that covariance depends on the local slope of $g$. In one dimension, with scalar estimate variance $\widehat v$, the formula is

W=\frac{g(\hat{\theta})^2}{\{g'(\hat{\theta})\}^2\widehat v}.

Thus the numerator is not enough: a given numerical departure $g(\hat{\theta})$ counts as more or less substantial depending on how rapidly the restriction changes with the parameter. For a Gaussian covariance matrix, a smooth restriction can be expressed using coordinates such as the distinct entries of $\Sigma$. Positive definiteness places the covariance parameter in an open set, so ordinary derivatives apply at interior covariance matrices.

Worked Example: A Nonlinear Restriction on a Gaussian Mean

Let $X_1,\ldots,X_{25}$ be independent $N(\theta,1)$ observations. The unrestricted MLE is $\hat{\theta}=\bar X$, with variance $1/25=0.04$. Suppose the observed mean is $1.2$, and test the nonlinear null $H_0:\theta^2=1$. Write the restriction as $g(\theta)=\theta^2-1$; the null contains both $\theta=1$ and $\theta=-1$.

The observed restriction is $g(1.2)=1.2^2-1=0.44$, and its derivative at the estimate is $g'(1.2)=2(1.2)=2.4$. The estimated variance of the restriction, by local linearization, is $$ \{g'(1.2)\}^2(0.04)=2.4^2(0.04)=0.2304. $$

Therefore $$ W=\frac{0.44^2}{0.2304} =\frac{0.1936}{0.2304}\approx 0.8403. $$

The null is a single smooth restriction near either null point because $g'(\theta)=2\theta$ is nonzero at $\theta=1$ and $\theta=-1$. The usual asymptotic calibration is $\chi^2_1$. This example also shows why the derivative is evaluated at $\hat{\theta}$: it converts uncertainty in the mean estimate to local uncertainty in the nonlinear restriction.

Worked Example: A Determinant Restriction on Gaussian Covariance

Suppose $X_1,\ldots,X_n$ are independent $N_2(0,\Sigma)$ observations with known mean zero and unknown covariance. The Gaussian MLE is $\hat{\Sigma}=n^{-1}\sum_i X_iX_i^\mathsf{T}$. For coordinates $(\sigma_{11},\sigma_{22},\sigma_{12})$, test the smooth null $H_0:\det(\Sigma)=1$, or $g(\Sigma)=\sigma_{11}\sigma_{22}-\sigma_{12}^2-1=0$.

Take $n=100$ and suppose the MLE is $$ \hat{\Sigma}=\begin{pmatrix}2&0.5\\0.5&1\end{pmatrix}. $$ This matrix is positive definite because its diagonal entries are positive and its determinant is $2(1)-0.5^2=1.75>0$. The restriction estimate is $g(\hat{\Sigma})=1.75-1=0.75$, and the gradient in the stated coordinate order is $(\sigma_{22},\sigma_{11},-2\sigma_{12})$, which at the estimate is $(1,2,-1)$.

For a Gaussian sample with known mean, the covariance formula for the entries of this MLE is $\operatorname{Cov}(\hat{\sigma}_{ij},\hat{\sigma}_{kl})=(\sigma_{ik}\sigma_{jl}+\sigma_{il}\sigma_{jk})/n$. Plugging in $\hat{\Sigma}$, the covariance estimate for $(\hat{\sigma}_{11},\hat{\sigma}_{22},\hat{\sigma}_{12})$ is $$ \widehat V=\frac{1}{100} \begin{pmatrix} 8&0.5&2\\ 0.5&2&1\\ 2&1&2.25 \end{pmatrix}. $$ For example, the first diagonal entry is $2\hat{\sigma}_{11}^2/100=8/100$, and the covariance between the first and third entries is $2\hat{\sigma}_{11}\hat{\sigma}_{12}/100=2/100$. The estimated variance of the restriction is $$ (1,2,-1)\widehat V(1,2,-1)^\mathsf{T} =\frac{8+8+2.25+2-4-4}{100} =0.1225. $$

The Wald statistic is consequently $$ W=\frac{0.75^2}{0.1225} =\frac{0.5625}{0.1225}\approx 4.592. $$

This is a one-restriction Wald statistic, conventionally calibrated against $\chi^2_1$ asymptotically. The covariance matrix used here is a plug-in estimate, and the delta-method variance is a local approximation; the displayed calculation does not assert an exact finite-sample chi-square law for this nonlinear statistic.

Chi-Square Calibration

The general calibration follows from asymptotic normality and a first-order Taylor expansion. State the conditions explicitly: the true parameter $\theta_0$ is an interior point satisfying $g(\theta_0)=0$; $g$ is continuously differentiable near $\theta_0$; the $q$ rows of $G(\theta_0)$ are linearly independent; and the estimator and its covariance estimate satisfy $\sqrt n(\hat{\theta}-\theta_0)\xrightarrow{d}N_d(0,V)$ and $n\widehat V\xrightarrow{p}V$, where $V$ is positive definite. These conditions include the usual regularity requirements for the likelihood estimator and its covariance estimate.

Theorem: Under the conditions just stated, the Wald statistic for $H_0:g(\theta)=0$ satisfies $W\xrightarrow{d}\chi^2_q$ when the null is true.

Expand the restriction.
Continuous differentiability gives $g(\hat{\theta})=G(\theta_0)(\hat{\theta}-\theta_0)+o_p(n^{-1/2})$, since $g(\theta_0)=0$ and $\hat{\theta}-\theta_0=O_p(n^{-1/2})$.

Find its limiting distribution.
Multiplying the expansion by $\sqrt n$ and applying the assumed normal limit gives $\sqrt n\,g(\hat{\theta})\xrightarrow{d}N_q(0,G_0VG_0^\mathsf{T})$, where $G_0=G(\theta_0)$. Full row rank of $G_0$ and positive definiteness of $V$ make this covariance nonsingular.

Pass to the quadratic form.
Consistency of $\hat{\theta}$, continuity of $G$, and $n\widehat V\xrightarrow{p}V$ imply $nG(\hat{\theta})\widehat V G(\hat{\theta})^\mathsf{T}\xrightarrow{p}G_0VG_0^\mathsf{T}$. Slutsky's theorem then gives the quadratic form of a standardized $q$-dimensional normal vector, which is $\chi^2_q$. $\square$

There is also an exact algebraic invariance worth using when writing restrictions. If $C$ is any nonsingular $q$ by $q$ matrix and the restriction is rewritten as $\tilde g(\theta)=Cg(\theta)$, then $\tilde G(\theta)=CG(\theta)$. The resulting Wald statistic is exactly unchanged wherever its inverses exist:

(Cg)^\mathsf{T}(C G\widehat V G^\mathsf{T}C^\mathsf{T})^{-1}(Cg) =g^\mathsf{T}(G\widehat V G^\mathsf{T})^{-1}g.

The equality follows from $(CMC^\mathsf{T})^{-1}=C^{-\mathsf{T}}M^{-1}C^{-1}$ for nonsingular $C$ and nonsingular $M$. Thus rescaling or taking nonsingular linear combinations of the same restrictions does not change the statistic. A general nonlinear rewriting need not preserve the finite-sample statistic, even when it describes the same null set.

Common Pitfalls and Practical Meaning

Using the wrong covariance scale. The matrix in the Wald denominator estimates uncertainty in $\hat{\theta}$, which usually decreases as $1/n$. Omitting that factor can change the statistic by a factor of $n$.
Ignoring the Jacobian. For nonlinear constraints, using only $g(\hat{\theta})$ ignores how sensitive the restriction is to changes in the parameter. The delta-method covariance is $G(\hat{\theta})\widehat V G(\hat{\theta})^\mathsf{T}$.
Counting parameters instead of restrictions. The reference degrees of freedom are $q$, the number of locally independent restrictions, not the dimension $d$ of the full parameter.
Assuming every null representation is harmless. Nonsingular linear transformations preserve the statistic, but nonlinear transformations can change its finite-sample value and may even have a zero gradient at a null point. The full-rank Jacobian condition is essential to the usual calibration.
Treating asymptotic calibration as exact. Except in special Gaussian cases such as the known-covariance mean test, the chi-square reference is generally an approximation justified by large-sample conditions.

Key takeaway: A Wald test evaluates the unrestricted estimate's departure from a null restriction relative to its estimated uncertainty. For smooth nonlinear restrictions, the Jacobian supplies the local conversion from parameter uncertainty to restriction uncertainty; regularity and full rank determine whether the resulting statistic has the stated chi-square calibration.

Check Your Understanding

Use the definitions and derivations above to answer the following.

For $H_0:A\mu=b$, what determines the chi-square degrees of freedom, and what covariance matrix is used for $A\bar X-b$?
Why is the Jacobian evaluated at $\hat{\theta}$ in the Wald statistic for a nonlinear restriction?
In the determinant example, identify the three covariance coordinates and the gradient of the restriction at the MLE.
State the asymptotic conditions that ensure a $\chi^2_q$ limit for a Wald statistic.
Explain why replacing $g$ by $Cg$ for a nonsingular matrix $C$ leaves the statistic unchanged, while a nonlinear rewriting may not.