Higher-Order Limits When the Gradient Vanishes

Why a Vanishing Gradient Changes the Limit

The ordinary delta method translates small estimation error through the slope of a transformation. If that slope is zero at the target, however, the linear approximation contributes nothing at the usual scale. The next nonzero Taylor term then controls both the rate and the shape of the limiting distribution.

For example, if $\hat\theta$ fluctuates around $\theta$ at order $n^{-1/2}$, a nonzero quadratic term produces transformed fluctuations of order $n^{-1}$, not $n^{-1/2}$. A cubic term produces order $n^{-3/2}$. The resulting limits are powers of the original normal limit, so they are generally not normal.

Setup: Suppose $\sqrt n(\hat\theta_n-\theta)\xrightarrow{d}Z$, where $Z\sim N(0,\tau^2)$ and $0<\tau^2<\infty$. Let $g$ be a scalar function that is $r$ times continuously differentiable in a neighborhood of $\theta$, for an integer $r\geq 2$. Assume $g^{(j)}(\theta)=0$ for $j=1,\ldots,r-1$, and $g^{(r)}(\theta)\neq 0$.

The Higher-Order Delta Method

When the derivatives below order $r$ vanish, Taylor expansion begins at order $r$. The coefficient of that term includes the factorial $r!$; it must not be omitted when calculating the limit.

Theorem: Under the setup above, $$ n^{r/2}\bigl(g(\hat\theta_n)-g(\theta)\bigr) \xrightarrow{d} \frac{g^{(r)}(\theta)}{r!}Z^r. $$ If $Z$ is nondegenerate, the limit is nondegenerate. Thus $n^{r/2}$ is the natural scaling for the transformed error.

The theorem concerns the transformation centered at its target value $g(\theta)$. It does not assert that the transformed estimator is centered at its finite-sample expectation; that is a different centering question.

Proof of the Higher-Order Limit

Write $\Delta_n=\hat\theta_n-\theta$ and $H_n=\sqrt n\,\Delta_n$. The assumed convergence in distribution implies that $H_n=O_p(1)$, and hence $\Delta_n=H_n/\sqrt n\xrightarrow{p}0$. Taylor's theorem with a remainder of smaller order gives $$ g(\theta+h)-g(\theta) = \frac{g^{(r)}(\theta)}{r!}h^r+R(h), \qquad \frac{R(h)}{|h|^r}\longrightarrow 0 \quad (h\longrightarrow 0,\ h\neq 0). $$ For $h=0$, take $R(0)=0$.

We verify that the scaled random remainder vanishes in probability. For any $\varepsilon>0$, there is a $\delta>0$ such that $|R(h)|\leq\varepsilon |h|^r$ whenever $|h|<\delta$. Therefore, on the event $|\Delta_n|<\delta$, $$ n^{r/2}|R(\Delta_n)| \leq \varepsilon n^{r/2}|\Delta_n|^r = \varepsilon |H_n|^r. $$ Since $H_n=O_p(1)$, its $r$th power is $O_p(1)$; and since $\Delta_n\xrightarrow{p}0$, the event $|\Delta_n|<\delta$ has probability tending to one. These facts imply $n^{r/2}R(\Delta_n)=o_p(1)$: for any fixed positive threshold, first bound $|H_n|^r$ by a sufficiently large constant with arbitrarily high probability, then choose $\varepsilon$ small enough that the displayed bound is below the threshold.

The Taylor expansion now yields $$ n^{r/2}\bigl(g(\hat\theta_n)-g(\theta)\bigr) = \frac{g^{(r)}(\theta)}{r!}H_n^r+o_p(1). $$ The map $x\mapsto x^r$ is continuous, so the continuous mapping theorem gives $H_n^r\xrightarrow{d}Z^r$. Slutsky's theorem then gives the asserted limit. Since a nondegenerate normal variable has probability zero of equaling zero, $Z^r$ is nonzero with probability one; multiplying it by the nonzero coefficient preserves nondegeneracy. $\square$

How the Order Determines the Shape

The limit is a constant times a power of a normal random variable. The parity of $r$ matters. If $r$ is even, $Z^r\geq 0$; the limit is supported on the nonnegative half-line when the Taylor coefficient is positive, and on the nonpositive half-line when it is negative. If $r$ is odd, the power preserves the sign of $Z$, so the limit has probability on both sides of zero. In either case, for $r\geq 2$ the limit is generally not normal.

For an even-order term with $Z\sim N(0,\tau^2)$, one can also write $Z^r=\tau^r U^r$, where $U\sim N(0,1)$. In particular, $Z^2$ has the distribution $\tau^2\chi^2_1$. Higher even powers have the corresponding powered chi-square form; they are not chi-square variables unless the power is exactly two.

Worked Example: A Quadratic Transformation of a Bernoulli Mean

Let $X_1,\ldots,X_n$ be independent Bernoulli random variables with success probability $1/2$, and let $\hat\theta_n=\bar X_n$. Find the limiting distribution of $n\,g(\bar X_n)$ for $g(x)=(x-1/2)^2$.

The mean and variance of each observation are $1/2$ and $1/4$, respectively. The central limit theorem gives $$ \sqrt n(\bar X_n-1/2)\xrightarrow{d}Z, \qquad Z\sim N(0,1/4). $$ Here $g(1/2)=0$, $g'(x)=2(x-1/2)$, so $g'(1/2)=0$, and $g''(1/2)=2$. The first nonzero Taylor term has order $r=2$, with coefficient $g''(1/2)/2!=1$. Thus $$ n\,g(\bar X_n) = n(\bar X_n-1/2)^2 \xrightarrow{d} Z^2. $$ Writing $Z=U/2$ for $U\sim N(0,1)$, the limit is $U^2/4$, or equivalently $\frac14\chi^2_1$. It is nonnegative and is not a normal distribution.

Worked Example: A Fourth-Order Transformation

Let $X_1,\ldots,X_n$ be independent $N(0,1)$ observations, let $\bar X_n$ estimate $\theta=0$, and set $g(x)=x^4$. Find the appropriate scaling and limit.

Since $\sqrt n\,\bar X_n\sim N(0,1)$ for every $n$, it converges in distribution to $Z\sim N(0,1)$. At $\theta=0$, the first three derivatives of $g$ vanish, while $g^{(4)}(0)=24$. The first nonzero Taylor coefficient is $24/4!=1$, so the theorem gives $$ n^2\bar X_n^4 = \bigl(\sqrt n\,\bar X_n\bigr)^4 \xrightarrow{d} Z^4. $$ Because $Z^2\sim\chi^2_1$, this limit can also be written as $(\chi^2_1)^2$, meaning the square of a chi-square-one random variable. It is not itself a chi-square variable.

Worked Example: A Cubic Term and an Odd Limit

Suppose $X_1,\ldots,X_n$ are independent $N(1,9)$ observations and $\bar X_n$ estimates $\theta=1$. Define $$ g(x)=e^{x-1}-1-(x-1)-\frac{(x-1)^2}{2}. $$ Find the rate and limiting distribution of $g(\bar X_n)$.

Set $u=x-1$. Then $g(1)=0$. Differentiating with respect to $u$, the first derivative at $u=0$ is $1-1-0=0$, and the second derivative at $u=0$ is $1-1=0$. The third derivative is $e^u$, so $g^{(3)}(1)=1$. Thus the first nonzero term is cubic, with coefficient $1/3!=1/6$.

Since the observations have variance $9$, $\sqrt n(\bar X_n-1)\sim N(0,9)$ for every $n$. Let $Z\sim N(0,9)$. The higher-order limit is $$ n^{3/2}g(\bar X_n)\xrightarrow{d}\frac{Z^3}{6}. $$ Equivalently, if $U\sim N(0,1)$ and $Z=3U$, the limit is $\frac{27}{6}U^3=\frac92 U^3$. Unlike the even-power limits above, this cubic limit can be positive or negative.

Why This Is the Natural Rate

The input error $\hat\theta_n-\theta$ is of order $n^{-1/2}$. If the first surviving change in $g$ is proportional to the $r$th power of that error, then the transformed error is of order $(n^{-1/2})^r=n^{-r/2}$. The theorem makes this rate calculation precise. When its limit is nondegenerate, using a smaller scaling factor makes the limit collapse to zero; using a substantially larger factor does not produce a finite nondegenerate limit.

This rate can be much faster than the familiar $n^{-1/2}$ rate for the original estimator. That does not mean the original estimator has become more precise: it means the particular transformation is locally flat, so small changes in its input have a smaller effect near $\theta$.

Common Mistake

A common error is to apply the first-order delta method and report a zero limit because $g'(\theta)=0$. For the quadratic example, this reasoning would use $\sqrt n\{g(\bar X_n)-g(1/2)\}$ and multiply the input limit by the zero derivative. That calculation only says the first-order term vanishes at the usual scale; it does not identify the first nonzero transformed fluctuation.

The correction is to inspect successive Taylor terms at $\theta$. If the first nonzero derivative is order $r$, use scaling $n^{r/2}$, include the coefficient $g^{(r)}(\theta)/r!$, and transform the normal limit by taking its $r$th power.

Key takeaway: When the first $r-1$ derivatives vanish and the $r$th does not, a root-$n$ input limit produces the transformed limit $\frac{g^{(r)}(\theta)}{r!}Z^r$ at rate $n^{r/2}$. The order determines the rate; the coefficient and the power determine the limiting law.

Check Your Understanding

For each question, identify the first nonzero Taylor order before deciding on a rate or limit.

If $\sqrt n(\hat\theta_n-\theta)\xrightarrow{d}Z$ and $g'(\theta)=0$, $g''(\theta)=6$, what is the limiting distribution of $n\{g(\hat\theta_n)-g(\theta)\}$?
Why does a fourth-order first nonzero term require scaling by $n^2$, rather than $n^{1/2}$ or $n$?
If $Z\sim N(0,\tau^2)$, what support does the limit have when the first nonzero term is even-order with a negative coefficient?
For a cubic first nonzero term, why can the limiting variable take both positive and negative values?
What error is made by concluding that the transformed limit is zero solely because $g'(\theta)=0$?