Deriving the Wald Statistic from a Quadratic Likelihood

Why a Quadratic Likelihood Leads to a Wald Test

A fitted GLMM reports parameter estimates together with an estimated covariance matrix. A Wald test combines these quantities to measure how far an estimate lies from a null hypothesis, relative to its uncertainty. The connection to likelihood is local: near a well-behaved maximum, the log likelihood is approximately a quadratic function, and the curvature of that quadratic determines the standard errors.

This derivation concerns a regular hypothesis about model parameters. In a GLMM, the likelihood for the model parameters is ordinarily the marginal likelihood: it accounts for the random effects by integrating them out. The random-effects structure and other model specifications are treated as fixed for the test. The derivation does not make the Wald test exact in a small sample; its reference distribution is an asymptotic approximation.

A linear null hypothesis can be written \(H_0:R\theta=r\), where \(R\) is a matrix with \(q\) linearly independent rows and \(r\) is a specified vector. The estimate’s discrepancy from the null is \(a=R\hat{\theta}-r\). Its estimated covariance is \(R\widehat V R^\mathsf{T}\), so the Wald statistic is the squared discrepancy measured in units of that covariance:

For a single scalar parameter \(\theta_j\) tested against \(\theta_{j,0}\), this reduces to \(W=(\hat{\theta}_j-\theta_{j,0})^2/\widehat{\mathrm{Var}}(\hat{\theta}_j)\), the square of the familiar estimate-minus-null divided by its standard error. For \(q\) independent restrictions, the regular large-sample reference distribution under \(H_0\) is chi-square with \(q\) degrees of freedom.

Derivation from the Local Likelihood Shape

At an interior maximum, the score (the gradient of the log likelihood) is zero: \(\nabla\ell(\hat{\theta})=0\). A second-order Taylor expansion around \(\hat{\theta}\) therefore has no first-order term:

This says that, locally, moving away from the maximum reduces the log likelihood by an amount determined by the information matrix. To see the test statistic in this shape, consider the point on the null hypothesis that is closest to \(\hat{\theta}\) under the quadratic distance defined by \(J(\hat{\theta})\). Let \(d=\hat{\theta}-\theta_0\) for a candidate null point \(\theta_0\). Then \(Rd=a\), and the quadratic distance is \(d^\mathsf{T}J(\hat{\theta})d\).

Minimizing this distance subject to \(Rd=a\) gives the constrained quadratic approximation. With a Lagrange multiplier \(\lambda\), differentiate \(d^\mathsf{T}Jd+2\lambda^\mathsf{T}(Rd-a)\) with respect to \(d\). The stationary point satisfies \(Jd+R^\mathsf{T}\lambda=0\), so \(d=-J^{-1}R^\mathsf{T}\lambda\). Substituting into the constraint gives \(-RJ^{-1}R^\mathsf{T}\lambda=a\). Hence:

The equality uses \(\widehat V=J^{-1}\). Thus the Wald statistic is the quadratic likelihood penalty for moving from the unrestricted maximum to the closest point satisfying the null, within the local quadratic approximation. This is a derivation of the Wald statistic, not a claim that a real GLMM likelihood is exactly quadratic or that the approximation gives the exact finite-sample test.

The chi-square calibration follows from the regular large-sample approximation \(\hat{\theta}\) being approximately multivariate normal under the null, with covariance estimated by \(\widehat V\). Applying \(R\) gives an approximately \(q\)-dimensional normal discrepancy with covariance \(R\widehat V R^\mathsf{T}\). The squared standardized discrepancy then has an approximate \(\chi^2_q\) distribution.

Worked Calculations

Conditions, Diagnostics, and Alternatives

The derivation and its chi-square calibration require regularity. In practical terms, the model likelihood should be sufficiently smooth near the maximum; the relevant parameters should be identifiable and at an interior point of their parameter space; and the information matrix should be finite and nonsingular. The large-sample approximation also requires an appropriate asymptotic regime. For clustered data, the number of independent clusters is often important; a large total number of observations does not by itself guarantee accurate inference if only a few clusters were observed.

For a GLMM, these conditions concern the likelihood actually fitted, including its random-effects distribution and any numerical approximation used to evaluate the marginal likelihood. Before relying on a Wald result, check that the fit converged, that the estimates are finite, and that the reported covariance matrix is stable and positive definite. Very large standard errors, near-singular information, coefficients that change greatly with small fitting changes, or estimates near a parameter-space boundary are warning signs. Separation in a binary model can also produce unstable or divergent fixed-effect estimates, undermining the regular quadratic approximation.

The Wald test is convenient because it needs an estimate and its covariance matrix, but its result depends on the local curvature at the unrestricted estimate. A likelihood-ratio test instead compares maximized log likelihoods under the unrestricted and null-constrained models. A profile-likelihood approach evaluates likelihood over values of the parameter of interest while re-estimating nuisance parameters. These methods use likelihood information differently, so their results need not match, especially when the likelihood is asymmetric or the sample is limited. The Wald test is most persuasive when the fit is regular and the likelihood is close to quadratic over the relevant range.