Comparing First- and Second-Order Approximations

Why Compare Approximation Orders?

A transformed sequence can behave on different scales depending on the derivatives of the transformation at the point where the original sequence is centered. If the first derivative is nonzero, a small change in the input usually produces a first-order change in the output. If that derivative vanishes, the quadratic term may instead determine the scale and limiting distribution. The practical question is not simply whether a Taylor expansion exists, but which term remains visible at the scale being studied.

Write $X_n=\theta+\delta_n$, where $\delta_n$ is small, and let $g$ be a real-valued function defined near $\theta$. For a positive sequence $r_n\to0$, the notation $\delta_n=O_p(r_n)$ means that $\delta_n/r_n$ is bounded in probability; $\delta_n=o_p(r_n)$ means that $\delta_n/r_n\xrightarrow{p}0$. In particular, if $\delta_n/r_n\xrightarrow{d}Z$ for a finite random variable $Z$, then $\delta_n=O_p(r_n)$.

Definition: A first-order approximation retains $g'(\theta)\delta_n$ in the expansion of $g(\theta+\delta_n)-g(\theta)$. A second-order approximation retains both $g'(\theta)\delta_n$ and $\tfrac12 g''(\theta)\delta_n^2$. Which approximation controls a limit depends on the derivatives at $\theta$, the rate of $\delta_n$, and the scale used for the transformed sequence.

Taylor Remainders and Stochastic Rates

Assume that $g$ is twice continuously differentiable on an open interval containing $\theta$, and that $P(X_n\text{ lies in that interval})\to1$. Taylor's formula with a Peano remainder gives, as $h\to0$,

g(\theta+h)-g(\theta) = g'(\theta)h+\frac12 g''(\theta)h^2+o(h^2).

The remainder statement is deterministic: after dividing the remainder by $h^2$, the result tends to zero as $h\to0$. When $h=\delta_n=O_p(r_n)$ and $r_n\to0$, this yields a stochastic expansion. In particular, $\delta_n\xrightarrow{p}0$, and the remainder is $o_p(r_n^2)$. The proof below explains why the deterministic remainder can be used for a random input.

Theorem: Suppose $g$ is twice continuously differentiable near $\theta$, $r_n\to0$, and $\delta_n=O_p(r_n)$. Then $$ g(\theta+\delta_n)-g(\theta) = g'(\theta)\delta_n+\frac12g''(\theta)\delta_n^2+o_p(r_n^2). $$ If also $\delta_n/r_n\xrightarrow{d}Z$, then:

When $g'(\theta)\ne0$, $r_n^{-1}\{g(\theta+\delta_n)-g(\theta)\}\xrightarrow{d}g'(\theta)Z$.
When $g'(\theta)=0$, $r_n^{-2}\{g(\theta+\delta_n)-g(\theta)\}\xrightarrow{d}\tfrac12g''(\theta)Z^2$.

In the second conclusion, a nondegenerate quadratic limit requires $g''(\theta)\ne0$ and $Z^2$ not to be almost surely zero.

Proof. Define $R(h)=g(\theta+h)-g(\theta)-g'(\theta)h-\tfrac12g''(\theta)h^2$. The Taylor expansion says $R(h)/h^2\to0$ as $h\to0$, with the ratio defined to be zero at $h=0$. Since $\delta_n=O_p(r_n)$ and $r_n\to0$, $\delta_n\xrightarrow{p}0$, so $R(\delta_n)/\delta_n^2\xrightarrow{p}0$. Also $\delta_n^2/r_n^2=O_p(1)$. Their product therefore converges to zero in probability, giving $R(\delta_n)=o_p(r_n^2)$. Events on which the Taylor expansion is not applicable have probability tending to zero by the assumed domain condition.

If $g'(\theta)\ne0$, divide the expansion by $r_n$. The quadratic term is $O_p(r_n^2)/r_n=O_p(r_n)=o_p(1)$, and the remainder is also $o_p(1)$. Thus the scaled transformed difference equals $g'(\theta)\delta_n/r_n+o_p(1)$, giving the first limit by convergence in distribution and Slutsky's theorem. If $g'(\theta)=0$, divide instead by $r_n^2$. The result is $\tfrac12g''(\theta)(\delta_n/r_n)^2+o_p(1)$. The continuous mapping theorem applied to the square function, followed by Slutsky's theorem, gives the second limit. $\square$

Worked Comparisons

Worked Example: A Logarithm with Nonzero Slope

Let $Z_n$ take the values $1$ and $-1$ with equal probability, and set $X_n=1+Z_n/\sqrt n$. For $n>1$, $X_n>0$, so $g(X_n)=\log X_n$ is defined. Here $\theta=1$, $r_n=n^{-1/2}$, and $g'(1)=1$, $g''(1)=-1$. Since $Z_n$ has the same distribution for every $n$, $Z_n\xrightarrow{d}Z$, where $Z$ is also equally likely to be $1$ or $-1$.

The first-order approximation is $\log(1+Z_n/\sqrt n)\approx Z_n/\sqrt n$. More precisely, the theorem gives

\sqrt n\log X_n\xrightarrow{d}Z.

The second-order expansion also identifies the error in that first-order approximation:

n\left[\log X_n-\frac{Z_n}{\sqrt n}\right] \xrightarrow{d}-\frac12Z^2=-\frac12.

For a numerical check at $n=100$, when $Z_n=1$ the scaled correction is $100[\log(1.1)-0.1]=-0.468982\ldots$, approximately $-0.46900$. When $Z_n=-1$, it is $100[\log(0.9)+0.1]=-0.536051\ldots$, approximately $-0.53605$. The corrections are not exactly $-1/2$ at finite $n$, but both approach $-1/2$ as $n$ increases. The first derivative determines the $n^{1/2}$ scale; curvature describes the smaller error at the $n$ scale.

Worked Example: A Quadratic Transformation at a Stationary Point

Let $Z\sim N(0,1)$, $X_n=Z/\sqrt n$, and $g(x)=x^2$, with $\theta=0$. Then $\delta_n=X_n$, $r_n=n^{-1/2}$, $g'(0)=0$, and $g''(0)=2$. The first-order term is identically zero, so it cannot describe the leading change in $g(X_n)$. The second-order expansion is exact:

g(X_n)-g(0)=X_n^2=\frac{Z^2}{n}.

Consequently,

n\{g(X_n)-g(0)\}=Z^2.

This equality holds for every $n$, not just asymptotically. The limit is the square of a standard normal random variable, with a chi-square distribution on one degree of freedom. In this case the derivative check changes the appropriate scale from $n^{1/2}$ to $n$, and the second-order term provides the nontrivial limit.

Worked Example: A Nonzero Slope with a Visible Curvature Correction

Let $Z_n$ again be equally likely to equal $1$ or $-1$, set $X_n=Z_n/\sqrt n$, and take $g(x)=x+x^2$ at $\theta=0$. Now $g'(0)=1$ and $g''(0)=2$. Direct substitution gives

g(X_n)=\frac{Z_n}{\sqrt n}+\frac{Z_n^2}{n}.

At the first-order scale,

\sqrt n\,g(X_n)=Z_n+\frac{Z_n^2}{\sqrt n} \xrightarrow{d} Z.

The quadratic contribution vanishes at this scale, so the first-order approximation controls the limiting distribution. But subtracting that approximation and using the finer scale reveals the correction:

n\left[g(X_n)-X_n\right]=Z_n^2=1.

Thus the first-order approximation correctly describes the leading behavior, while the second-order term exactly describes its error here. A term can be negligible for one limit and decisive for a more finely scaled comparison.

A Practical Derivative and Rate Check

For a given transformed sequence, the following checks separate the leading approximation from a correction term. First identify the center $\theta$ and a rate $r_n$ such that $(X_n-\theta)/r_n$ has a finite limiting distribution. Then evaluate $g'(\theta)$ and $g''(\theta)$. The resulting comparison is summarized here.

Derivative check	Leading term	Natural scale	Interpretation
$g'(\theta)\ne0$	$g'(\theta)\delta_n$	$r_n^{-1}$	First order controls the leading limit; the quadratic term is smaller by a factor of order $r_n$.
$g'(\theta)=0$, $g''(\theta)\ne0$	$\tfrac12g''(\theta)\delta_n^2$	$r_n^{-2}$	Second order controls the leading limit, provided the rescaled input has a nonzero limiting square.
$g'(\theta)=g''(\theta)=0$	Neither displayed term is informative	Not determined by these derivatives	Check higher derivatives or use a more direct expansion; second order alone may give only a zero limit.

The table describes the usual case $\delta_n=O_p(r_n)$ with $\delta_n/r_n$ converging in distribution. It does not say that every sequence has a nondegenerate limit at the listed scale. For example, the limit $Z$ may be zero with probability one, or a coefficient may vanish. A derivative check identifies a candidate leading term; the limiting input and its coefficient must still make that term nontrivial.

Common Mistakes and Why the Comparison Matters

Using the second-order scale just because a second derivative exists. If $g'(\theta)\ne0$, the linear term is generally of order $r_n$, while the quadratic term is of order $r_n^2$. The second-order term is a correction, not the leading limit at scale $r_n^{-1}$.
Calling a zero first-order limit the answer. When $g'(\theta)=0$, scaling by $r_n^{-1}$ may produce a zero limit. That does not show that the transformed difference is negligible at every scale; if $g''(\theta)\ne0$, the $r_n^{-2}$ scale can reveal its behavior.
Assuming a nonzero second derivative whenever the first vanishes. If both $g'(\theta)$ and $g''(\theta)$ are zero, the second-order expansion does not identify a nonzero leading term. A higher-order check may be needed.
Ignoring the remainder condition. A Taylor polynomial alone is not a stochastic limit argument. The input must approach the expansion point, the function must have the needed smoothness near it, and the remainder must be negligible at the scale used.

These comparisons matter whenever a transformed estimator or statistic is centered and rescaled to obtain an asymptotic distribution. The derivative check tells which rate is plausible, while the remainder check justifies discarding the terms that are smaller at that rate. For greater accuracy, the same expansion can also quantify the error left by the leading approximation.

Key takeaway: Compare the orders of $g'(\theta)\delta_n$ and $\tfrac12g''(\theta)\delta_n^2$ at the proposed scale. A nonzero slope gives the usual first-order limit; a vanishing slope with nonzero curvature makes the quadratic term the leading candidate. If both derivatives vanish, do not infer the relevant rate from a second-order expansion alone.

Check Your Understanding

For each question, identify which derivative and scale determine the comparison, and explain any condition needed for a nontrivial limit.

Suppose $X_n-\theta=O_p(n^{-1/2})$, $(X_n-\theta)\sqrt n\xrightarrow{d}Z$, and $g'(\theta)\ne0$. At what scale does the first-order transformed difference have a limit?
Under the same input assumptions, suppose $g'(\theta)=0$ and $g''(\theta)\ne0$. Write the candidate limiting variable at the appropriate scale.
For $g(x)=\cos x$ at $\theta=0$, determine whether the first or second derivative supplies the leading term. What does this imply for the usual $n^{1/2}$ input rate?
Why does a Taylor remainder that is $o_p(r_n^2)$ suffice for a first-order limit at scale $r_n^{-1}$ when $r_n\to0$?
If $g'(\theta)=g''(\theta)=0$, what conclusion can and cannot be drawn from a second-order expansion?

Derivative check	Leading term	Natural scale	Interpretation
\(g'(\theta)\ne0\)	\(g'(\theta)\delta_n\)	\(r_n^{-1}\)	First order controls the leading limit; the quadratic term is smaller by a factor of order \(r_n\).
\(g'(\theta)=0\), \(g''(\theta)\ne0\)	\(\tfrac12g''(\theta)\delta_n^2\)	\(r_n^{-2}\)	Second order controls the leading limit, provided the rescaled input has a nonzero limiting square.
\(g'(\theta)=g''(\theta)=0\)	Neither displayed term is informative	Not determined by these derivatives	Check higher derivatives or use a more direct expansion; second order alone may give only a zero limit.