Parameterization Dependence of Wald Tests

Why Parameterization Matters for Wald Tests

A statistical model can often describe the same unknown feature using different parameters. For a probability $p$, for instance, one could instead use its logit, $\eta=\log\{p/(1-p)\}$. The model and the null hypothesis have not changed, but a Wald test can give a different finite-sample statistic depending on which parameter is used. Understanding this dependence is important when interpreting a test or comparing software that reports tests on different scales.

The source of the difference is that a Wald statistic measures a squared distance from the null, scaled by an estimated variance. Under a nonlinear transformation, both the distance and the variance change; the usual variance transformation is local, using a derivative at the estimate. The resulting ratio is generally not unchanged. By contrast, score and likelihood-ratio tests are constructed from the likelihood and the null model itself, so a one-to-one change of coordinates leaves their values unchanged.

Definition: For a scalar parameter $\theta$, testing $H_0:\theta=\theta_0$, the Wald statistic is $$ W_\theta=\frac{(\hat{\theta}-\theta_0)^2}{\widehat{\operatorname{Var}}(\hat{\theta})}, $$ where the estimated variance is evaluated using the stated Wald convention, commonly the inverse observed or expected information at the unrestricted estimate. If $\eta=h(\theta)$ is a smooth one-to-one reparameterization, the corresponding Wald statistic tests $H_0:\eta=h(\theta_0)$ using $\hat{\eta}=h(\hat{\theta})$ and its estimated variance.

For a smooth transformation with nonzero derivative, the delta-method variance estimate is $\widehat{\operatorname{Var}}(\hat{\eta})=[h'(\hat{\theta})]^2\widehat{\operatorname{Var}}(\hat{\theta})$. The transformed Wald statistic is therefore

W_\eta= \frac{\{h(\hat{\theta})-h(\theta_0)\}^2} {[h'(\hat{\theta})]^2\widehat{\operatorname{Var}}(\hat{\theta})}.

A nonlinear function difference $h(\hat{\theta})-h(\theta_0)$ is not generally equal to $h'(\hat{\theta})(\hat{\theta}-\theta_0)$. That mismatch explains why the two ratios need not agree. A linear transformation has a constant derivative and does preserve the scalar Wald statistic.

Worked Example: A Binomial Wald Test on the Probability and Logit Scales

Suppose $X\sim\operatorname{Binomial}(10,p)$, and the observed count is $x=2$. Test $H_0:p=0.5$. The maximum-likelihood estimate is $\hat p=2/10=0.2$. Use the inverse expected information evaluated at the unrestricted estimate for the Wald variances.

For the probability parameter, the estimated variance is $\hat p(1-\hat p)/n=(0.2)(0.8)/10=0.016$. Thus

W_p=\frac{(0.2-0.5)^2}{0.016} =\frac{0.09}{0.016} =5.625.

Now use the logit parameter $\eta=\log\{p/(1-p)\}$. The null value is $\eta_0=0$, and $\hat\eta=\log(0.2/0.8)=\log(0.25)\approx-1.386294$. The logit-scale variance estimate is $1/\{n\hat p(1-\hat p)\}=1/(10\cdot0.16)=0.625$. Consequently,

W_\eta=\frac{(-1.386294-0)^2}{0.625} \approx 3.075.

Both tests use the same data, model, and null hypothesis, yet their statistics differ: $5.625$ versus approximately $3.075$. This is finite-sample parameterization dependence, not a change in the scientific hypothesis.

Likelihood-Ratio Invariance

Let a likelihood be $L(\theta)$, with unrestricted parameter space $\Theta$ and null parameter set $\Theta_0$. The likelihood-ratio statistic is $2\{\sup_{\theta\in\Theta}\ell(\theta)-\sup_{\theta\in\Theta_0}\ell(\theta)\}$, where $\ell$ is the log-likelihood. If $\eta=h(\theta)$ is a one-to-one reparameterization, both parameter sets are carried to corresponding sets, and every likelihood value is the same model likelihood expressed at a different coordinate. No Jacobian factor is introduced: the likelihood is a function of the parameter, not a density for the parameter.

Theorem: Suppose $\eta=h(\theta)$ is a one-to-one mapping from $\Theta$ onto $\mathcal{H}$, and the transformed null set is $\mathcal{H}_0=\{h(\theta):\theta\in\Theta_0\}$. When the same likelihood model is expressed in either parameterization, the likelihood-ratio statistic for $\Theta_0$ equals that for $\mathcal{H}_0$, exactly for every data set.

Proof. Define the transformed log-likelihood by $\ell_\eta(\eta)=\ell_\theta(h^{-1}(\eta))$. Because $h$ maps $\Theta$ onto $\mathcal{H}$, its unrestricted likelihood values satisfy

\sup_{\eta\in\mathcal{H}}\ell_\eta(\eta) =\sup_{\eta\in\mathcal{H}}\ell_\theta(h^{-1}(\eta)) =\sup_{\theta\in\Theta}\ell_\theta(\theta).

The same reasoning restricted to the corresponding null sets gives $\sup_{\eta\in\mathcal{H}_0}\ell_\eta(\eta)=\sup_{\theta\in\Theta_0}\ell_\theta(\theta)$. Subtracting these equal quantities and multiplying by two proves exact equality of the statistics. $\square$

Worked Example: Likelihood-Ratio Invariance for a Binomial Model

Continue with $n=10$, $x=2$, and $H_0:p=0.5$. Ignoring the binomial coefficient, the log-likelihood is $\ell(p)=2\log p+8\log(1-p)$. The unrestricted estimate is $\hat p=0.2$, so the likelihood-ratio statistic is

2\{\ell(0.2)-\ell(0.5)\} =2\left\{2\log(0.2/0.5)+8\log(0.8/0.5)\right\} \approx 3.855.

The null on the logit scale is $\eta=0$, and the unrestricted logit estimate corresponds exactly to $\hat p=0.2$. The likelihood values at those points are the same values just calculated, so the logit-scale likelihood-ratio statistic is also approximately $3.855$. Unlike the Wald statistic, it does not depend on whether $p$ or its logit is used.

Score-Test Invariance

A score test evaluates the likelihood's local slope at the null, scaled by information at the null. Under a smooth change of coordinates, the score changes by a derivative factor, and the information changes by the square of that factor. In the standardized quadratic statistic, those changes cancel. This statement presumes a regular, locally one-to-one reparameterization and the same null model; for composite restrictions, the score and information are taken in the tested directions after accounting for nuisance parameters.

Result: In a regular scalar model with no nuisance parameter, let $\eta=h(\theta)$, with differentiable $h$ and $h'(\theta_0)\ne0$. The score statistic for testing $\theta=\theta_0$ has the same value in the $\theta$ and $\eta$ parameterizations.

Proof. Write $U_\theta=\partial\ell/\partial\theta$ for the score and $I_\theta$ for Fisher information. The chain rule gives $U_\eta=U_\theta\,d\theta/d\eta$. The information transforms as $I_\eta=I_\theta(d\theta/d\eta)^2$, by the information transformation rule for a regular change of scalar coordinates. The score statistic in the $\eta$ coordinate is therefore

\frac{U_\eta^2}{I_\eta} =\frac{U_\theta^2(d\theta/d\eta)^2} {I_\theta(d\theta/d\eta)^2} =\frac{U_\theta^2}{I_\theta}.

The derivative is nonzero, so cancellation is valid. This proves equality of the statistics. $\square$

Worked Example: Three Tests in the Same Binomial Model

For the binomial data $n=10$, $x=2$, and null $p_0=0.5$, the score statistic is

S=\frac{(x-np_0)^2}{np_0(1-p_0)} =\frac{(2-5)^2}{10(0.5)(0.5)} =\frac{9}{2.5} =3.6.

This calculation is on the probability scale. On the logit scale, the chain-rule factor multiplying the score is $dp/d\eta=p(1-p)$, evaluated at the null as $0.25$. The information is multiplied by $0.25^2$, so the score-squared-over-information ratio remains $3.6$. The likelihood-ratio statistic is approximately $3.855$, as shown above. Thus the three values are: probability-scale Wald $5.625$, score $3.6$, and likelihood ratio $3.855$. Re-expressing the Wald test on the logit scale changes its value to approximately $3.075$, while the score and likelihood-ratio values remain fixed.

Why Wald Tests Become Equivalent Asymptotically

Finite-sample disagreement does not contradict the usual first-order asymptotic equivalence of Wald statistics under regularity conditions. Suppose $\hat{\theta}\xrightarrow{p}\theta_0$, $h$ is continuously differentiable near $\theta_0$, and $h'(\theta_0)\ne0$. The mean value theorem gives $h(\hat{\theta})-h(\theta_0)=h'(\tilde{\theta})(\hat{\theta}-\theta_0)$ for a point $\tilde{\theta}$ between $\hat{\theta}$ and $\theta_0$. Consistency implies $\tilde{\theta}\xrightarrow{p}\theta_0$, hence $h'(\tilde{\theta})/h'(\hat{\theta})\xrightarrow{p}1$. The delta-method variance transformation then yields

\frac{W_\eta}{W_\theta} = \left\{\frac{h'(\tilde{\theta})}{h'(\hat{\theta})}\right\}^{\!2} \xrightarrow{p}1,

whenever the ratio is defined. Therefore the two statistics agree to first order under the null, even though they need not be numerically equal for a given sample. In regular models, this is why different Wald parameterizations can share the same asymptotic chi-square calibration without being the same finite-sample test.

Worked Example: A Normal Mean and an Exponential Parameter

Let $X_1,\ldots,X_4$ be independent $N(\mu,1)$ observations with $\bar X=1$. Test $H_0:\mu=0$. In the mean parameterization, the estimated variance of $\hat\mu=\bar X$ is $1/4$, so the Wald statistic is

W_\mu=\frac{(1-0)^2}{1/4}=4.

Now define $\eta=e^\mu$, so the same null is $H_0:\eta=1$, and the unrestricted estimate is $\hat\eta=e^{\bar X}=e$. The inverse information variance estimate on the $\eta$ scale is $\hat\eta^2/4=e^2/4$, since the derivative of $\mu=\log\eta$ is $1/\eta$. The transformed Wald statistic is

W_\eta=\frac{(e-1)^2}{e^2/4} =4(1-e^{-1})^2 \approx 1.599.

The statistics differ substantially. But the likelihood-ratio statistic is $4(\bar X-0)^2=4$ in either coordinate, and the score statistic is also $4$. This example makes clear that a nonlinear Wald calculation measures a coordinate-dependent distance, whereas the other two procedures retain their likelihood-based values.

Common Pitfall and Practical Interpretation

Incorrect reasoning: “A one-to-one transformation does not change the parameter or the null, so every test statistic must be identical.” A one-to-one map does preserve which distributions and hypotheses are being considered, but it does not preserve the numerical distance used by a Wald statistic. The correction is to distinguish the test's null hypothesis from its chosen coordinate representation: the Wald statistic is generally coordinate-dependent at finite sample size, while the likelihood-ratio statistic is exactly invariant and the regular score statistic is invariant under smooth reparameterization.

Another practical issue is reporting only “the Wald test” without naming the parameter scale or variance convention. For nonlinear restrictions or several parameters, the restriction representation and its Jacobian also matter. A Wald test is useful and often asymptotically well behaved, but its finite-sample result should be interpreted with awareness of these choices. For this reason, a difference between two Wald outputs need not indicate a coding error or different underlying data; it may reflect a different, yet equivalent, parameterization.

Key takeaway: Wald tests use a local variance to scale a parameter-space distance, so nonlinear reparameterization can change their finite-sample values. Under regularity, corresponding Wald tests are first-order equivalent; score and likelihood-ratio tests are invariant under smooth one-to-one changes of coordinates for the same null model.

Check Your Understanding

Try these questions before moving on.

For a scalar transformation $\eta=h(\theta)$, write the delta-method variance estimate for $\hat\eta$. Which feature of a nonlinear $h$ permits the Wald statistic to change?
In the binomial example, why does the logit-scale likelihood-ratio statistic equal the probability-scale likelihood-ratio statistic?
State the score and information transformation rules under a smooth scalar reparameterization, and explain why their factors cancel in the score statistic.
What regularity condition on $h$ is needed for the first-order comparison of the two Wald statistics?
Does a different finite-sample Wald value necessarily mean that the null hypothesis or the underlying model has changed? Explain.