Wald Tests with Nuisance Parameters

Why Nuisance Parameters Belong in a Wald Test

A hypothesis may concern only one parameter, or a small group of parameters, even though a model estimates many others. The parameters named in the hypothesis are the target parameters; the remaining parameters are nuisance parameters. “Nuisance” does not mean irrelevant: estimating them can affect the uncertainty of the target estimates.

For a Wald test, the relevant uncertainty is the covariance of the target estimator when the model’s parameters are estimated jointly. It is obtained by taking the target block of the full estimated covariance matrix. In general, it is not the variance that would apply if nuisance parameters were known and held fixed.

If the estimated covariance of \(\hat\theta\) is reported directly, take its rows and columns corresponding to \(\psi\). If instead a procedure reports an estimate \(\hat V\) of the asymptotic covariance of \(\sqrt n(\hat\theta-\theta)\), then the estimated covariance of \(\hat\theta\) is \(\hat V/n\), and its target block is \(\hat V_{\psi\psi}/n\). Keeping track of this scaling prevents a common factor-of-\(n\) error.

The Wald Statistic and Its Null Calibration

For a \(q\)-dimensional target, a large value of \(W\) indicates that the estimate is far from the null value relative to its estimated uncertainty. Under regularity conditions, the null reference distribution is chi-square with \(q\) degrees of freedom. The result below states the conditions needed for that calibration.

Proof. Let \(Z_n=\sqrt n(\hat\psi-\psi_0)\), the target coordinates of \(\sqrt n(\hat\theta-\theta_0)\), and let \(V_{\psi\psi}\) be the corresponding block of \(V\). Marginal convergence gives \(Z_n\xrightarrow{d}N(0,V_{\psi\psi})\). Consistency of \(\hat V\) implies \(\hat V_{\psi\psi}\xrightarrow{p}V_{\psi\psi}\). Rewrite the statistic as

Because \(V_{\psi\psi}\) is positive definite, matrix inversion is continuous at \(V_{\psi\psi}\), so \(\hat V_{\psi\psi}^{-1}\xrightarrow{p}V_{\psi\psi}^{-1}\). Slutsky’s theorem and continuity of the quadratic-form map give the limiting distribution of \(Z_n^\mathsf{T}V_{\psi\psi}^{-1}Z_n\). If \(Z\sim N(0,V_{\psi\psi})\), then \(V_{\psi\psi}^{-1/2}Z\sim N(0,I_q)\); its squared Euclidean length has a \(\chi^2_q\) distribution. Therefore \(W\xrightarrow{d}\chi^2_q\). \(\square\)

At significance level \(\alpha\), reject when \(W\) exceeds the \(1-\alpha\) quantile of \(\chi^2_q\). Equivalently, the large-sample p-value is the upper-tail probability \(P(\chi^2_q\ge W)\). This calibration is asymptotic; the theorem does not assert that the statistic has an exact chi-square distribution in a finite sample.

Why the Full Covariance Matrix Matters

In a regular likelihood model, the covariance block used in the Wald statistic can be understood through the information matrix. Let the per-observation Fisher information for \(\theta=(\psi^\mathsf{T},\lambda^\mathsf{T})^\mathsf{T}\) be partitioned as

When nuisance parameters are estimated, the asymptotic covariance of the full maximum likelihood estimator is \(I^{-1}/n\). Thus the target covariance is the \(\psi\psi\) block of \(I^{-1}/n\). It is generally not \(I_{\psi\psi}^{-1}/n\), which corresponds to using the target information as if the nuisance parameters were fixed.

Proof. Write the first block column of \(I^{-1}\) as \(\begin{pmatrix}A\\B\end{pmatrix}\), so that

The lower block equation gives \(I_{\lambda\psi}A+I_{\lambda\lambda}B=0\), hence \(B=-I_{\lambda\lambda}^{-1}I_{\lambda\psi}A\). Substituting this into the upper block equation yields

The matrix in parentheses is invertible because \(I\) is positive definite. Therefore \(A\) equals its inverse. Since \(A\) is the upper-left block of \(I^{-1}\), the identity follows. \(\square\)

The subtracted term reflects the information about the target that is entangled with the nuisance parameters. When the target and nuisance scores are not orthogonal, estimating the nuisance parameters changes the target’s uncertainty. The block-inverse identity makes precise why one must use the full information matrix before extracting the target covariance.

Worked Calculations

Common Pitfall and Practical Checks

Wrong reasoning: “The null concerns only \(\psi\), so I should use the variance for \(\hat\psi\) with \(\lambda\) held fixed.” This changes the estimation problem. In the actual fit, \(\lambda\) is estimated, and its uncertainty and association with \(\hat\psi\) are reflected in the target block of the full covariance matrix.

Correction: Fit the model with nuisance parameters estimated as specified, obtain the full covariance matrix for that fit, and extract the target rows and columns. Then invert that target block to form the Wald statistic. For multiple target parameters, keep their off-diagonal covariances; for any target dimension, check whether the reported matrix is the covariance of \(\hat\theta\) or the asymptotic covariance of \(\sqrt n(\hat\theta-\theta)\).

The main practical checks are: the null value and target coordinates have the same dimension; the extracted covariance block is symmetric and positive definite; and the inverse is applied to the entire target block, not separately to its entries. The number of chi-square degrees of freedom here is the number \(q\) of target coordinates in \(H_0:\psi=\psi_0\).