Second-Order Bias From Nonlinear Transformations

Why Curvature Changes the Center

A transformation can make an estimator easier to interpret or can reduce how much its uncertainty varies across parameter values. But transforming an estimator also changes its center. If \(T_n\) fluctuates around \(\theta\), a curved function \(g\) does not generally transform those fluctuations symmetrically around \(g(\theta)\). The resulting discrepancy is a form of bias.

The first-order delta method describes the leading variance of \(g(T_n)\) using the slope \(g'(\theta)\). To understand its leading bias, the slope alone is not enough: the second derivative matters. The sign of the curvature determines whether the leading correction pushes the transformed estimator upward or downward.

Definition: The bias of an estimator \(U_n\) for a target \(\psi\) is \(E[U_n]-\psi\), when this expectation exists. For a transformed estimator \(g(T_n)\) targeting \(g(\theta)\), its bias is \(E[g(T_n)]-g(\theta)\). A second-order bias approximation is an approximation to this expectation obtained by expanding \(g\) around \(\theta\) through its quadratic term.

The Second-Order Bias Formula

Write \(\Delta_n=T_n-\theta\). A Taylor expansion separates the transformed error into a linear part, a quadratic part, and a remainder:

g(T_n)-g(\theta) = g'(\theta)\Delta_n +\frac{1}{2}g''(\theta)\Delta_n^2 +R_n.

Taking expectations makes the roles of the terms visible. The linear term reflects any bias already present in \(T_n\). The quadratic term reflects the spread of \(T_n\), weighted by the curvature of \(g\). In particular, \(E[\Delta_n^2]=\operatorname{Var}(T_n)+\{E[\Delta_n]\}^2\).

Theorem (second-order bias expansion): Suppose \(E|T_n-\theta|^2<\infty\), \(E[g(T_n)]\) exists, \(g\) is twice differentiable at \(\theta\), and the Taylor remainder satisfies \(E[R_n]=o(n^{-1})\), where \(R_n=g(T_n)-g(\theta)-g'(\theta)(T_n-\theta)-\frac12g''(\theta)(T_n-\theta)^2\). Suppose also that \(E[T_n-\theta]=O(n^{-1})\) and \(\operatorname{Var}(T_n)=O(n^{-1})\). Then

E[g(T_n)]-g(\theta) = g'(\theta)E[T_n-\theta] +\frac{1}{2}g''(\theta)\operatorname{Var}(T_n) +o(n^{-1}).

Proof. By the definition of \(R_n\), take expectations in the Taylor identity. The quadratic term has expectation \(E[(T_n-\theta)^2]=\operatorname{Var}(T_n)+\{E[T_n-\theta]\}^2\). Therefore

\begin{aligned} E[g(T_n)]-g(\theta) ={}&g'(\theta)E[T_n-\theta] +\frac12 g''(\theta)\operatorname{Var}(T_n)\\ &+\frac12g''(\theta)\{E[T_n-\theta]\}^2+E[R_n]. \end{aligned}

The assumed order \(E[T_n-\theta]=O(n^{-1})\) implies \(\{E[T_n-\theta]\}^2=O(n^{-2})=o(n^{-1})\), and the assumed remainder expectation is \(o(n^{-1})\). Combining these terms proves the stated expansion. \(\square\)

The remainder condition is important: pointwise Taylor expansion alone does not justify taking expectations or discarding the remainder. One useful sufficient condition is that \(g\) has a bounded third derivative on a range containing \(T_n\) and \(\theta\), and \(E|T_n-\theta|^3=o(n^{-1})\). Taylor's theorem then gives \(|R_n|\leq M|T_n-\theta|^3/6\), where \(M\) bounds \(|g'''|\), so \(E|R_n|=o(n^{-1})\). For sample means of independent, identically distributed variables with finite fourth moment, the third absolute central moment of the sample mean is \(O(n^{-3/2})\): by Cauchy–Schwarz it is at most \(\{E(T_n-\theta)^2 E(T_n-\theta)^4\}^{1/2}\), and those two moments are \(O(n^{-1})\) and \(O(n^{-2})\), respectively.

When \(T_n\) is unbiased, the first term in the theorem vanishes. The leading bias is then \(\frac12g''(\theta)\operatorname{Var}(T_n)\). Positive curvature gives a positive correction; negative curvature gives a negative one, provided the variance is positive. If \(T_n\) itself has bias of order \(n^{-1}\), its contribution \(g'(\theta)E[T_n-\theta]\) is of the same order and must not be omitted.

Worked Calculations

Worked Example: A Logarithmic Transformation of an Exponential Mean

Let \(X_1,\ldots,X_n\) be independent exponential observations with mean \(\theta>0\), and let \(T_n=\bar X\). Then \(E[T_n]=\theta\) and \(\operatorname{Var}(T_n)=\theta^2/n\). Consider \(g(t)=\log(1+t)\), which is defined for \(t\geq0\).

The derivatives at the target are \(g'(\theta)=1/(1+\theta)\), \(g''(\theta)=-1/(1+\theta)^2\), and \(g'''(t)=2/(1+t)^3\). The third derivative is bounded on \(t\geq0\). The exponential distribution has finite fourth moment, so the sample-mean moment argument above gives \(E|T_n-\theta|^3=O(n^{-3/2})\); thus the remainder is \(o(n^{-1})\).

Since \(T_n\) is unbiased, the first-order bias term is zero. The second-order approximation is

E[\log(1+\bar X)]-\log(1+\theta) \approx \frac12\left(-\frac{1}{(1+\theta)^2}\right)\frac{\theta^2}{n} = -\frac{\theta^2}{2n(1+\theta)^2}.

For \(\theta=2\) and \(n=100\), this gives \(-4/(2\cdot100\cdot9)=-1/450\), about \(-0.00222\). The negative sign follows from the concavity of the logarithm: the transformed sample mean is biased downward relative to \(\log(1+\theta)\), to leading order.

Worked Example: A Square-Root Transformation of a Poisson Rate

Suppose \(X_1,\ldots,X_n\) are independent Poisson variables with mean \(\lambda>0\), and \(T_n=\bar X\). Then \(E[T_n]=\lambda\) and \(\operatorname{Var}(T_n)=\lambda/n\). Use the transformation \(g(t)=\sqrt{1+t}\), defined for \(t\geq0\).

Here \(g''(t)=-1/[4(1+t)^{3/2}]\), and \(g'''(t)=3/[8(1+t)^{5/2}]\), which is bounded over the range of \(T_n\). Poisson variables have finite fourth moment, so the third absolute central moment of \(\bar X\) is \(O(n^{-3/2})\). The theorem applies, and unbiasedness again removes the linear bias term:

E[\sqrt{1+\bar X}]-\sqrt{1+\lambda} \approx \frac12\left(-\frac{1}{4(1+\lambda)^{3/2}}\right)\frac{\lambda}{n} = -\frac{\lambda}{8n(1+\lambda)^{3/2}}.

At \(\lambda=3\) and \(n=60\), the approximation is \(-3/[8\cdot60\cdot 8]=-1/1280\), about \(-0.000781\). The correction is small here, but it is not zero. Its size depends on both the variance of the estimator and the curvature at the target.

Worked Example: A Quadratic Transformation of a Binomial Proportion

Let \(Y_1,\ldots,Y_n\) be independent Bernoulli variables with success probability \(p\), and let \(T_n=\bar Y\). Then \(E[T_n]=p\) and \(\operatorname{Var}(T_n)=p(1-p)/n\). For \(g(t)=t^2\), the target is \(p^2\), while \(g'(p)=2p\) and \(g''(p)=2\).

The second-order formula gives

E[T_n^2]-p^2 \approx \frac12(2)\frac{p(1-p)}{n} = \frac{p(1-p)}{n}.

In this case the result is exact, not merely approximate: \(E[T_n^2]=\operatorname{Var}(T_n)+\{E[T_n]\}^2=p(1-p)/n+p^2\). For \(p=0.4\) and \(n=25\), the bias is \(0.4(0.6)/25=0.0096\), and \(E[T_n^2]=0.16+0.0096=0.1696\). The positive bias matches the positive curvature \(g''=2\).

What Variance Stabilization Does—and Does Not—Do

A variance-stabilizing transformation is chosen to make the leading variance of a transformed estimator approximately constant across parameter values. In a common scalar setting, if \(\operatorname{Var}(T_n)\) is approximately \(v(\theta)/n\), the first-order delta method gives

\operatorname{Var}(g(T_n)) \approx \{g'(\theta)\}^2\frac{v(\theta)}{n}.

Choosing \(g'(\theta)\) proportional to \(1/\sqrt{v(\theta)}\) makes this expression constant to first order. But this condition concerns the derivative \(g'\) and the variance. It does not set \(g''(\theta)\) to zero, nor does it remove any bias already in \(T_n\). Consequently, it does not generally remove the second-order bias.

For example, in the Poisson calculation above, \(T_n\) is unbiased but the transformed estimator has a negative correction because \(g''(\lambda)<0\). The purpose of a variance-stabilizing transformation and the purpose of bias correction are distinct. The first targets variation across parameter values; the second targets the difference between an estimator's expectation and its target. Neither property, by itself, guarantees the other.

Key takeaway: To approximate the bias of \(g(T_n)\), include both the transformed bias inherited from \(T_n\), through \(g'(\theta)E[T_n-\theta]\), and the curvature correction, through \(\frac12g''(\theta)\operatorname{Var}(T_n)\). Check that the expected Taylor remainder is negligible at the order being reported.

Common Pitfall

A tempting but incomplete argument is: “\(T_n\) is unbiased, so \(g(T_n)\) is unbiased for \(g(\theta)\).” This silently assumes that expectation commutes with a nonlinear transformation. In general, it does not. For the quadratic binomial example, \(E[T_n]=p\), but \(E[T_n^2]=p^2+p(1-p)/n\), not \(p^2\). The correction is exactly the variance of \(T_n\).

Another mistake is to report only the curvature term when \(T_n\) has its own bias. If \(E[T_n-\theta]\) is of order \(n^{-1}\), then \(g'(\theta)E[T_n-\theta]\) is also of order \(n^{-1}\) and contributes at the same leading order. Finally, a Taylor expansion is not automatically an expectation approximation: the remainder must be controlled in expectation, not merely shown to be small for each fixed realization near \(\theta\).

Check Your Understanding

Use the expansion and examples above to answer these questions.

If \(T_n\) is unbiased and \(g''(\theta)>0\), what is the sign of the leading curvature correction when \(\operatorname{Var}(T_n)>0\)?
For a biased \(T_n\) with bias of order \(n^{-1}\), which two terms contribute to the leading bias of \(g(T_n)\)?
Why is a pointwise Taylor expansion not, by itself, enough to justify the second-order bias formula?
For a Poisson sample mean, explain why the square-root example has a negative leading bias correction.
Does choosing a transformation to stabilize first-order variance generally make the transformed estimator unbiased? Explain briefly.