Second-Order Delta Method

Why a Second-Order Approximation Is Needed

The first-order delta method translates an asymptotic distribution through a smooth function by multiplying by its derivative. If that derivative is zero at the limiting parameter, the first-order approximation vanishes. It does not follow that the transformed estimator has no variation; rather, its leading variation may be determined by the curvature of the function. In that case the relevant scale is often \(n\), not \(\sqrt n\), and the limit is generally a quadratic form rather than a normal random variable.

This tutorial treats a scalar-valued function of a vector estimator. The essential calculation is a second-order Taylor expansion, followed by continuous mapping and Slutsky’s theorem. The asymptotic normal vector supplies the random input to the quadratic form; the Hessian describes how the function curves at the limiting parameter.

Statement of the Second-Order Delta Method

Definition: Let \(\theta\in\mathbb{R}^d\), and let \(g\) be a real-valued function defined on an open neighborhood of \(\theta\). The gradient \(\nabla g(\theta)\) is the vector of first partial derivatives at \(\theta\), and the Hessian \(H_g(\theta)\) is the \(d\times d\) matrix of second partial derivatives at \(\theta\). If \(g\) is twice continuously differentiable on that neighborhood, its Hessian is symmetric.

Theorem (Second-Order Delta Method): Suppose \(T_n\in\mathbb{R}^d\), \(\theta\in\mathbb{R}^d\), and \[ \sqrt n(T_n-\theta)\xrightarrow{d} Z,\qquad Z\sim N(0,\Sigma), \] where \(\Sigma\) is a finite, symmetric, positive semidefinite covariance matrix. Suppose \(g\) is twice continuously differentiable on an open neighborhood of \(\theta\), and \(\nabla g(\theta)=0\). Then \[ n\{g(T_n)-g(\theta)\}\xrightarrow{d}\frac12 Z^{\mathsf T}H_g(\theta)Z. \]

The zero-gradient condition removes the linear term in Taylor’s formula. The factor \(1/2\) is part of the quadratic Taylor term and must be retained. The conclusion allows a singular covariance matrix and a degenerate limit; the quadratic form need not have a continuous distribution.

Proof of the Limit

Write \(h_n=T_n-\theta\) and \(H=H_g(\theta)\). The assumed convergence in distribution implies that \(\sqrt n h_n\) is bounded in probability, so \(h_n=O_p(n^{-1/2})\) and \(h_n\xrightarrow{p}0\). For all sufficiently small \(h\), Taylor’s formula with an integral remainder gives

g(\theta+h)-g(\theta) =\nabla g(\theta)^{\mathsf T}h +\int_0^1(1-t)\,h^{\mathsf T}H_g(\theta+th)h\,dt.

Because \(\nabla g(\theta)=0\), subtracting \(\frac12h^{\mathsf T}Hh\) leaves the remainder

R(h)=\int_0^1(1-t)\,h^{\mathsf T} \{H_g(\theta+th)-H\}h\,dt.

Use the Euclidean norm for vectors and its induced matrix norm. Continuity of the Hessian at \(\theta\) implies that \[ \omega(\delta)=\sup_{\|u\|\leq\delta}\|H_g(\theta+u)-H\| \] tends to zero as \(\delta\downarrow0\), taking \(\delta\) small enough that the ball lies in the domain of \(g\). The inequality \(|x^{\mathsf T}Ax|\leq\|A\|\|x\|^2\), together with \(\int_0^1(1-t)\,dt=1/2\), yields

|R(h)|\leq \frac12\|h\|^2\omega(\|h\|).

Since \(h_n\xrightarrow{p}0\), \(\omega(\|h_n\|)\xrightarrow{p}0\). Also \(n\|h_n\|^2=O_p(1)\), because \(\sqrt n h_n=O_p(1)\). The product rule for stochastic orders therefore gives \(nR(h_n)=o_p(1)\). Consequently,

n\{g(T_n)-g(\theta)\} =\frac12(\sqrt n h_n)^{\mathsf T}H(\sqrt n h_n)+o_p(1).

The map \(z\mapsto\frac12z^{\mathsf T}Hz\) is continuous. The continuous mapping theorem and then Slutsky’s theorem give the stated limit. \(\square\)

Reading the Quadratic-Form Limit

The theorem gives the limit directly in terms of \(Z\). A useful further description expresses it as a weighted sum of independent chi-squared variables. Let \(\Sigma^{1/2}\) be the symmetric positive semidefinite square root of \(\Sigma\). A normal vector with covariance \(\Sigma\) has the same distribution as \(\Sigma^{1/2}G\), where \(G\sim N(0,I_d)\). Define the symmetric matrix \[ A=\Sigma^{1/2}H_g(\theta)\Sigma^{1/2}. \] By the spectral theorem, there is an orthogonal matrix \(Q\) and real eigenvalues \(\lambda_1,\ldots,\lambda_d\) such that \(A=Q\,\operatorname{diag}(\lambda_1,\ldots,\lambda_d)Q^{\mathsf T}\). Orthogonal transformations preserve the standard multivariate normal distribution, so \(Q^{\mathsf T}G\) has independent standard normal coordinates. Thus the limiting variable has the same distribution as

\frac12G^{\mathsf T}AG \ \overset{d}{=}\ \frac12\sum_{j=1}^d\lambda_j U_j^2 =\frac12\sum_{j=1}^d\lambda_j\chi^2_{1,j},

where \(U_1,\ldots,U_d\) are independent \(N(0,1)\) variables and \(\chi^2_{1,j}=U_j^2\). To see the equality, substitute the spectral decomposition and set \(U=Q^{\mathsf T}G\). The eigenvalues may be positive, negative, or zero: the resulting limit can be nonnegative, indefinite, or degenerate. This representation is a description of the limit, not in general a single chi-squared distribution.

Worked Example: Squared Error of a Bernoulli Sample Proportion

Let \(X_1,\ldots,X_n\) be independent Bernoulli random variables with success probability \(p=0.3\), and let \(\hat p_n=n^{-1}\sum_{i=1}^n X_i\). Find the limit distribution of \(n(\hat p_n-p)^2\).

The central limit theorem gives \(\sqrt n(\hat p_n-p)\xrightarrow{d}Z\), where \(Z\sim N(0,p(1-p))=N(0,0.21)\). Use \(g(t)=(t-p)^2\), for which \(g(p)=0\), \(g'(t)=2(t-p)\), and \(g'(p)=0\). Its second derivative is \(g''(t)=2\), so the theorem gives

n(\hat p_n-p)^2 \xrightarrow{d}\frac12(2)Z^2=Z^2.

Writing \(Z=\sqrt{0.21}\,U\) for \(U\sim N(0,1)\), the limit is \(0.21U^2\), or \(0.21\chi^2_1\). The limit is nonnegative, as it must be because every term \(n(\hat p_n-p)^2\) is nonnegative.

Worked Example: A Quadratic Function of Two Sample Means

Suppose \(X_1,\ldots,X_n\) are independent \(N(0,1)\), \(Y_1,\ldots,Y_n\) are independent \(N(0,4)\), and the two samples are independent. Let \(T_n=(\bar X_n,\bar Y_n)^{\mathsf T}\) and \(g(x,y)=x^2+2y^2\). Find the limiting distribution of \(n g(T_n)\).

Here \(\theta=(0,0)^{\mathsf T}\), and \(\sqrt nT_n\) is exactly \(N(0,\Sigma)\) for every \(n\), with \(\Sigma=\operatorname{diag}(1,4)\). The gradient is \((2x,4y)^{\mathsf T}\), which vanishes at \(\theta\), and

H_g(\theta)= \begin{pmatrix} 2&0\\ 0&4 \end{pmatrix}.

The theorem gives the limit \(\frac12(Z_1,Z_2)H_g(\theta)(Z_1,Z_2)^{\mathsf T}=Z_1^2+2Z_2^2\), with independent \(Z_1\sim N(0,1)\) and \(Z_2\sim N(0,4)\). Set \(Z_1=U_1\) and \(Z_2=2U_2\) for independent standard normals. The limit is therefore

U_1^2+8U_2^2 \ \overset{d}{=}\ \chi^2_{1,1}+8\chi^2_{1,2},

where the two chi-squared variables are independent. In this example the result can also be checked directly: \(n\bar X_n^2=(\sqrt n\bar X_n)^2\) and \(2n\bar Y_n^2=2(\sqrt n\bar Y_n)^2\), and those scaled means have exactly the normal distributions used above.

Worked Example: A Product Under Correlated Normal Limits

Let each observation \((X_i,Y_i)^{\mathsf T}\) be independent and identically distributed as \(N(0,\Sigma)\), with \[ \Sigma=\begin{pmatrix}1&1/2\\1/2&1\end{pmatrix}, \] and let \(T_n=(\bar X_n,\bar Y_n)^{\mathsf T}\). For \(g(x,y)=xy\), find the limit of \(n\bar X_n\bar Y_n\) and express it as a weighted sum of independent chi-squared variables.

The multivariate central limit theorem gives \(\sqrt nT_n\xrightarrow{d}Z\sim N(0,\Sigma)\). At \(\theta=(0,0)^{\mathsf T}\), \(g(\theta)=0\), the gradient is zero, and

H_g(\theta)= \begin{pmatrix} 0&1\\ 1&0 \end{pmatrix}.

The second-order delta method first gives \(n\bar X_n\bar Y_n\xrightarrow{d}Z_1Z_2\). To find the weighted chi-squared form, use \(A=\Sigma^{1/2}H_g(\theta)\Sigma^{1/2}\). Its eigenvalues can be calculated from the similar matrix \(\Sigma H_g(\theta)\):

\Sigma H_g(\theta)= \begin{pmatrix} 1/2&1\\ 1&1/2 \end{pmatrix}, \qquad \det\!\left( \begin{pmatrix}1/2&1\\1&1/2\end{pmatrix} -\lambda I \right) =(1/2-\lambda)^2-1.

Setting the determinant to zero gives \(\lambda=3/2\) and \(\lambda=-1/2\). Including the theorem’s factor \(1/2\), the limit has distribution \[ \frac34\chi^2_{1,1}-\frac14\chi^2_{1,2}, \] with independent chi-squared variables. The negative weight is appropriate: a product can be negative, so its limiting distribution need not be supported only on nonnegative values.

Common Pitfall and Practical Interpretation

A frequent error is to keep the first-order \(\sqrt n\) scale after the gradient vanishes, or to omit the factor \(1/2\) from the Hessian term. For example, with \(g(t)=(t-p)^2\), the derivative at \(p\) is zero, so a first-order approximation predicts no nondegenerate \(\sqrt n\)-scaled limit. The correct calculation uses the second derivative \(2\): the limit for \(n\{g(\hat p_n)-g(p)\}\) is \(\frac12(2)Z^2=Z^2\), not \(2Z^2\). The scale and the factor both follow from the Taylor expansion.

In applications, the method explains why a transformed estimator can have a non-normal limit even when the original estimator is asymptotically normal. The Hessian and covariance matrix jointly determine that limit: curvature alone is not enough, since variability in directions with little or no asymptotic variance contributes little or not at all. The eigenvalue representation makes this dependence explicit, while also warning that an indefinite Hessian can produce a signed sum rather than an ordinary chi-squared variable.

Check Your Understanding

Use the theorem’s assumptions and Taylor expansion to answer these questions.

Why does \(\sqrt n(T_n-\theta)\xrightarrow{d}Z\) imply that the Taylor remainder, multiplied by \(n\), is \(o_p(1)\) when the Hessian is continuous near \(\theta\)?
For \(g(t)=(t-\theta)^2\), identify the gradient and Hessian at \(\theta\), then state the second-order limit in terms of \(Z\sim N(0,\sigma^2)\).
Can a second-order delta-method limit be negative? What feature of the Hessian determines whether negative values are possible?
Why is the limit in the two-dimensional product example not a single chi-squared random variable?
If one eigenvalue of \(\Sigma^{1/2}H_g(\theta)\Sigma^{1/2}\) is zero, what does its term contribute to the weighted chi-squared representation?

What You'll Learn