Deriving the Wald Test from Asymptotic Normality

From an Estimated Parameter to a Test of a Restriction

An unrestricted maximum-likelihood estimate does not usually satisfy a null hypothesis exactly. The Wald test asks whether its estimated departure from the null restriction is large compared with the sampling uncertainty in that departure. For a smooth restriction, the delta method supplies that uncertainty: it transforms the covariance of the parameter estimate into the covariance of the restriction estimate.

This construction applies to generalized linear models (GLMs) when the unrestricted estimator has a suitable asymptotic normal distribution and its covariance can be estimated consistently. The test uses the unrestricted fit; it does not require fitting the model subject to the null.

For example, a single restriction on two parameters might be \(h(\theta)=\theta_1^2-\theta_2=0\). Its Jacobian is the row vector \((2\theta_1,-1)\), which has rank one. The restriction is not required to be linear; differentiability and the rank condition are what allow a local normal approximation to be used.

Deriving the Wald Statistic

Suppose the unrestricted maximum-likelihood estimator \(\hat\theta\) is consistent and, at the null parameter, satisfies \(\sqrt{n}(\hat\theta-\theta_0)\xrightarrow{d}N(0,V)\), where \(V\) is positive definite. In likelihood applications, \(V\) is often the limit of \(n\) times the estimator’s covariance, for example the inverse of a suitable limiting information matrix. Assume also that an estimator \(\hat C_n\) of \(\operatorname{Var}(\hat\theta)\) satisfies \(n\hat C_n\xrightarrow{p}V\). These are asymptotic assumptions; the resulting test is not generally an exact finite-sample test.

Under \(H_0\), the delta method gives the limiting distribution of \(h(\hat\theta)\). Since the null value is zero, the transformed estimate itself is a measure of departure from the null. Its estimated covariance is obtained by applying the Jacobian to the estimated covariance of \(\hat\theta\).

Here \(H(\hat\theta)\hat C_nH(\hat\theta)^\mathsf{T}\) estimates the covariance of the \(q\)-vector \(h(\hat\theta)\). Its inverse exists with probability approaching one under the stated conditions. Equivalently, the statistic can be written with an explicit sample-size factor:

To see the equivalence, for any invertible matrix \(A\) and positive scalar \(n\), \((A/n)^{-1}=nA^{-1}\). Thus the two expressions differ only in whether the factor \(n\) is written inside or outside the inverse.

Why the Reference Distribution Is Chi-Square

The derivation has two steps. First, approximate the transformed estimate by a multivariate normal vector. Second, standardize its covariance and use the distribution of a squared length of independent standard normal variables.

The Taylor expansion in the first step is justified locally: differentiability at \(\theta_0\) means \(h(\hat\theta)=h(\theta_0)+H(\theta_0)(\hat\theta-\theta_0)+r_n\), where \(\|r_n\|/\|\hat\theta-\theta_0\|\xrightarrow{p}0\). Asymptotic normality implies \(\sqrt n(\hat\theta-\theta_0)=O_p(1)\), so \(\sqrt n r_n=o_p(1)\). This yields the claimed linearization without assuming that the restriction is exactly linear.

For completeness, \(A\) is positive definite because for any nonzero \(a\in\mathbb{R}^q\), full row rank of \(H(\theta_0)\) implies \(H(\theta_0)^\mathsf{T}a\ne0\). Hence \(a^\mathsf{T}Aa=(H(\theta_0)^\mathsf{T}a)^\mathsf{T}V(H(\theta_0)^\mathsf{T}a)>0\). By the spectral theorem, \(A\) has a symmetric positive-definite square root \(A^{1/2}\). The vector \(A^{-1/2}Z\) is therefore \(N(0,I_q)\), and \(Z^\mathsf{T}A^{-1}Z=(A^{-1/2}Z)^\mathsf{T}(A^{-1/2}Z)\), the sum of squares of \(q\) independent standard normal variables. That sum has a \(\chi^2_q\) distribution. Convergence of the covariance estimate and Slutsky’s theorem establish the theorem. \(\square\)

Worked Calculations

Conditions, Interpretation, and a Common Pitfall

The chi-square calibration is a large-sample result under the null. It relies on a regular, interior parameter value, asymptotic normality, a consistent covariance estimate, and a locally full-rank Jacobian. If the restriction is redundant, or the parameter lies on a boundary with a nonstandard estimator limit, these conditions may fail and the stated reference distribution need not apply. In a GLM with nuisance parameters, the covariance used must correspond to the full estimator or its appropriate marginal covariance; omitting uncertainty in nuisance parameters can give the wrong variance for \(h(\hat\theta)\).

A frequent mistake is to compare \(h(\hat\theta)^2\) directly with a chi-square distribution, as though every restriction estimate had variance one. The correction is to scale the departure by its estimated covariance, including the Jacobian transformation. For multiple restrictions, that means the full covariance matrix and its inverse, not separate unadjusted squared departures.

A second caution is that the Wald test is built around the unrestricted estimate and its local linear approximation. A nonlinear transformation can change the numerical Wald statistic, even when it describes the same null set. For that reason, the parameterization and the restriction function should be chosen deliberately, and the result should be understood as an asymptotic test rather than an invariant measure of evidence.