Non-Invariance of Wald Tests Under Reparameterization

Why Parameterization Matters for a Wald Test

A parameter can often be described on more than one scale. A probability $p$, for example, can be expressed as a log-odds $\beta=\log\{p/(1-p)\}$. These descriptions may encode exactly the same scientific hypothesis, but the Wald test measures a distance using an estimated standard error on the chosen scale. A nonlinear transformation changes both that distance and its estimated standard error, and the resulting finite-sample statistic need not be the same.

The likelihood-ratio test behaves differently: it compares maximized likelihoods over the null and alternative parameter spaces. A one-to-one change of coordinates changes how those spaces are labelled, not the likelihood values being compared. The distinction is important whenever a hypothesis can be stated on multiple scales.

Wald Statistics on Different Scales

Definition: Suppose $\hat\theta$ estimates a scalar parameter $\theta$, and $\widehat{\operatorname{Var}}(\hat\theta)$ estimates its variance. The Wald statistic for $H_0:\theta=\theta_0$ is $$ W_\theta=\frac{(\hat\theta-\theta_0)^2}{\widehat{\operatorname{Var}}(\hat\theta)}. $$ For a differentiable scalar function $g$, a Wald statistic for $H_0:g(\theta)=g_0$, using the delta-method variance estimate, is $$ W_g=\frac{\{g(\hat\theta)-g_0\}^2}{\{g'(\hat\theta)\}^2\widehat{\operatorname{Var}}(\hat\theta)}. $$

The derivative in the denominator converts uncertainty on the $\theta$-scale to estimated uncertainty on the $g(\theta)$-scale. But the numerator is the transformed estimate’s distance from the null, not a linearized distance. For a nonlinear $g$, that distance generally does not equal $g'(\hat\theta)(\hat\theta-\theta_0)$. This is the source of the finite-sample difference.

For a vector parameter $\theta$ and a smooth vector restriction $r(\theta)=0$ with $q$ components, the corresponding Wald statistic is

W_r=r(\hat\theta)^\mathsf{T} \left\{R(\hat\theta)\widehat{\operatorname{Var}}(\hat\theta)R(\hat\theta)^\mathsf{T}\right\}^{-1} r(\hat\theta), \qquad R(\theta)=\frac{\partial r(\theta)}{\partial\theta^\mathsf{T}}.

This formula requires the estimated covariance matrix to exist and the matrix inside the inverse to be nonsingular. In regular models, its rank is $q$, and under the null its large-sample reference distribution is $\chi^2_q$. The statistic itself, however, can depend on the particular restriction function used to express the null.

Worked Example: A Binomial Probability and Its Logit

Let $Y\sim\operatorname{Binomial}(20,p)$, and suppose the observed number of successes is $y=15$. Compare Wald tests of $H_0:p=0.5$ using $p$ and $\beta=\log\{p/(1-p)\}$. The hypotheses are equivalent because $p=0.5$ exactly when $\beta=0$.

The maximum-likelihood estimate is $\hat p=15/20=0.75$. The estimated variance on the probability scale is $\hat p(1-\hat p)/20$, so

\widehat{\operatorname{Var}}(\hat p)=\frac{0.75(0.25)}{20}=0.009375, \qquad W_p=\frac{(0.75-0.5)^2}{0.009375} =\frac{0.0625}{0.009375}=6.6667.

On the logit scale, $\hat\beta=\log(0.75/0.25)=\log 3\approx1.098612$. The binomial model’s estimated variance for the logit estimate is $1/\{20\hat p(1-\hat p)\}=1/3.75$. Thus

W_\beta =\frac{(\log 3)^2}{1/3.75} =3.75(\log 3)^2 \approx 4.5261.

The two Wald statistics differ, although the null hypothesis is the same. Both use a squared estimate-to-null distance divided by an estimated variance, but those distances are calculated on different scales.

Likelihood-Ratio Invariance

For a parameter space $\Theta$, a null subset $\Theta_0\subseteq\Theta$, and likelihood $L(\theta)$, define the likelihood-ratio statistic by

-2\log\Lambda =-2\log\left\{ \frac{\sup_{\theta\in\Theta_0}L(\theta)} {\sup_{\theta\in\Theta}L(\theta)} \right\}.

The suprema are used whether or not they are attained. Under a one-to-one reparameterization, each candidate parameter value corresponds to exactly one point in the new coordinates. Therefore the likelihood values and the sets over which the suprema are taken correspond exactly.

Theorem: Let $\phi=T(\theta)$ be a bijection from $\Theta$ onto $\Phi$, and let $\Phi_0=T(\Theta_0)$. If the reparameterized likelihood is $L_\phi(\phi)=L(T^{-1}(\phi))$, then the likelihood-ratio statistic for $\Theta_0$ equals the statistic for $\Phi_0$.

Proof. Because $T$ is a bijection, every $\theta\in\Theta$ is $T^{-1}(\phi)$ for exactly one $\phi\in\Phi$. Consequently,

\sup_{\phi\in\Phi}L_\phi(\phi) =\sup_{\phi\in\Phi}L(T^{-1}(\phi)) =\sup_{\theta\in\Theta}L(\theta).

The same argument restricted to the null sets gives

\sup_{\phi\in\Phi_0}L_\phi(\phi) =\sup_{\theta\in\Theta_0}L(\theta).

The numerator and denominator of the likelihood ratio are therefore unchanged, so their ratio and $-2\log\Lambda$ are unchanged. No Jacobian factor is introduced: this is a re-expression of a likelihood as a function of the parameter, not a change of variables in a probability density over parameter values. $\square$

Worked Example: The Same Binomial Likelihood-Ratio Test

Continue with 15 successes in 20 trials. Under the alternative, $\hat p=0.75$; under $H_0:p=0.5$, the fitted probability is $0.5$. The binomial coefficient cancels in the likelihood ratio, giving

-2\log\Lambda =2\left[15\log\left(\frac{0.75}{0.5}\right) +5\log\left(\frac{0.25}{0.5}\right)\right] =2[15\log(1.5)+5\log(0.5)] \approx 5.23248.

To three decimal places the statistic is $5.232$. In logit coordinates, the alternative estimate is $\log 3$ and the null value is $0$. The numerator and denominator likelihood suprema are the same two likelihood values as before, so the likelihood-ratio statistic remains $5.23248$, not either of the two Wald statistics.

Affine Reparameterizations Preserve Wald Statistics

There is an important contrast with nonlinear transformations. Suppose the coordinate change is affine and invertible, $\phi=A\theta+b$, where $A$ is nonsingular. If the null is transformed along with the coordinates, the usual quadratic Wald statistic is preserved. In particular, for a scalar restriction $c^\mathsf{T}\theta=d$, its expression in the new coordinates is $c^\mathsf{T}A^{-1}(\phi-b)=d$, which is still an affine restriction.

Theorem: Let $\hat\theta$ have estimated covariance matrix $\widehat V$, and test $H_0:C\theta=d$, where $C$ has full row rank. The Wald statistic $$ (C\hat\theta-d)^\mathsf{T}(C\widehat V C^\mathsf{T})^{-1}(C\hat\theta-d) $$ is unchanged under any invertible affine change of coordinates $\phi=A\theta+b$, provided the null restriction and covariance estimate are transformed consistently.

Proof. Under this change, $\hat\phi=A\hat\theta+b$ and $\widehat V_\phi=A\widehat V A^\mathsf{T}$. The restriction in the new coordinates is $C A^{-1}(\phi-b)=d$. Its estimated residual is

C A^{-1}(\hat\phi-b)-d =C A^{-1}A\hat\theta-d=C\hat\theta-d.

The covariance of that residual, calculated in the new coordinates, is

C A^{-1}\widehat V_\phi(A^{-1})^\mathsf{T}C^\mathsf{T} =C A^{-1}A\widehat V A^\mathsf{T}(A^{-1})^\mathsf{T}C^\mathsf{T} =C\widehat V C^\mathsf{T}.

Both the residual vector and its covariance matrix are thus unchanged. Substitution into the Wald quadratic form proves the claim. $\square$

Worked Example: A Linear Change of Two Coordinates

Suppose $\hat\theta=(2,1)^\mathsf{T}$, with estimated covariance matrix

\widehat V= \begin{pmatrix}4&1\\1&2\end{pmatrix},

and test $H_0:\theta_2=0$. The original Wald statistic is $1^2/2=0.5$. Now set $\phi_1=\theta_1+\theta_2$ and $\phi_2=\theta_1-\theta_2$. Then $\hat\phi=(3,1)^\mathsf{T}$, and the null becomes $\phi_1-\phi_2=0$.

The transformed covariance matrix is

\widehat V_\phi= \begin{pmatrix}1&1\\1&-1\end{pmatrix} \widehat V \begin{pmatrix}1&1\\1&-1\end{pmatrix}^{\mathsf T} = \begin{pmatrix}8&2\\2&4\end{pmatrix}.

The estimated contrast is $3-1=2$, and its estimated variance is $8+4-2(2)=8$. Hence the transformed Wald statistic is $2^2/8=0.5$, agreeing with the original statistic.

Asymptotic Agreement Does Not Mean Exact Invariance

For a smooth nonlinear transformation with nonzero derivative at the null value, the delta method explains why different Wald statistics often have the same first-order large-sample behavior. It does not make their finite-sample values identical. For example, if $\hat\theta$ is consistent and asymptotically normal, with $\sqrt n(\hat\theta-\theta_0)\xrightarrow{d}N(0,I(\theta_0)^{-1})$, and the estimated variance is consistent at the corresponding $1/n$ scale, then a differentiable $g$ with $g'(\theta_0)\ne0$ satisfies

g(\hat\theta)-g(\theta_0) =g'(\theta_0)(\hat\theta-\theta_0)+o_p(n^{-1/2}), \qquad g'(\hat\theta)=g'(\theta_0)+o_p(1).

The first relation follows from differentiability and $\hat\theta-\theta_0=O_p(n^{-1/2})$; the second follows from consistency and continuity of $g'$ at $\theta_0$. Squaring the first relation and dividing by the delta-method variance gives $W_g=W_\theta+o_p(1)$, under these regularity conditions. Thus both statistics have the same limiting $\chi^2_1$ null distribution. This first-order equivalence does not remove their finite-sample discrepancy.

Worked Example: Wald Tests for a Poisson Rate

Let $Y_1,\ldots,Y_5$ be independent Poisson observations with common rate $\lambda$, and suppose their total is 20. Then $\hat\lambda=20/5=4$. Test $H_0:\lambda=2$, equivalently $H_0:\eta=\log\lambda=\log 2$.

The estimated variance of $\hat\lambda$ is $\hat\lambda/5=0.8$, so

W_\lambda=\frac{(4-2)^2}{0.8}=5.

On the log-rate scale, $\hat\eta=\log 4$, with estimated variance $1/(5\hat\lambda)=1/20=0.05$. Therefore

W_\eta=\frac{(\log 4-\log 2)^2}{0.05} =20(\log 2)^2\approx9.609.

The likelihood-ratio statistic compares the Poisson likelihood at $\hat\lambda=4$ with its value at $\lambda=2$:

-2\log\Lambda =2\left[20\log\left(\frac{4}{2}\right)-5(4-2)\right] =40\log 2-20 \approx7.726.

Replacing $\lambda$ by the one-to-one coordinate $\eta=\log\lambda$ does not change either likelihood maximum, so the likelihood-ratio statistic remains approximately $7.726$. The two Wald statistics differ because the log transformation is nonlinear.

Common Pitfall and Practical Interpretation

Wrong reasoning: “The null hypotheses are equivalent, so their Wald statistics must be equal.” Equivalence identifies the same parameter values under the null; it does not make nonlinear distances or variance estimates equal.
Correction: Check which parameter scale and restriction function define the Wald statistic. A nonlinear transformation can change its finite-sample value; an invertible affine coordinate change preserves the usual quadratic statistic when the restriction and covariance are transformed consistently.
Do not infer exact agreement from asymptotic equivalence. Smooth, locally invertible transformations typically give first-order equivalent Wald tests under regularity conditions, but their observed statistics can differ appreciably in finite samples.
Do not confuse Wald and likelihood-ratio invariance. The likelihood-ratio statistic is exactly invariant under a bijective reparameterization because it compares the same likelihood suprema. The Wald statistic is built from an estimated distance and covariance on a specified scale.

In GLMs, this distinction matters when reporting tests for probabilities, odds, rates, or regression effects expressed through different links. State the scale on which a Wald test is calculated. If exact invariance to a one-to-one re-expression is required, the likelihood-ratio statistic has that property; it does not follow merely from both tests having the same asymptotic reference distribution.

Check Your Understanding

Use the definitions and derivations above to answer these questions.

Why can Wald tests of $p=p_0$ and $\log\{p/(1-p)\}=\log\{p_0/(1-p_0)\}$ differ in a finite sample?
What two likelihood suprema are compared by a likelihood-ratio test, and why are they unchanged under a bijective reparameterization?
For an invertible affine coordinate change, what must happen to the null restriction and covariance estimate for the Wald statistic to be preserved?
Under what local regularity conditions do Wald statistics on smoothly transformed scalar scales have the same first-order asymptotic behavior?
In the binomial example, which of the three reported statistics is exactly unchanged by switching from $p$ to its logit, and why?