Stochastic Expansions for Estimators

Why Linearize an Estimator?

An estimator may be a nonlinear function of sample quantities or the solution to a nonlinear sample equation. A first-order stochastic expansion expresses its error as a simpler leading term plus a remainder that is negligible at a specified rate. This separates the main source of sampling variation from higher-order effects, while making clear what assumptions justify ignoring those effects.

The expansion itself does not establish an asymptotic distribution. It identifies a leading random term; a distributional conclusion also requires a limit result for that term. Here the focus is on the linearization and on checking the order of its remainder.

Definition: Let $\hat{\theta}_n$ estimate a scalar or vector quantity with probability limit $\theta_0$, and let $r_n>0$ be a deterministic sequence tending to zero. A first-order stochastic expansion at rate $r_n$ has the form $$ \hat{\theta}_n-\theta_0=L_n+o_p(r_n), $$ where $L_n=O_p(r_n)$ is the leading random term. For random variables or vectors, $X_n=O_p(r_n)$ means $X_n/r_n$ is bounded in probability; $X_n=o_p(r_n)$ means $X_n/r_n$ converges in probability to zero. In particular, $o_p(r_n)$ is negligible relative to the target rate $r_n$, whereas $O_p(r_n)$ need not be.

The common rate $r_n=n^{-1/2}$ arises when sample averages have fluctuations of that order, but an expansion can be stated at another rate. The rate must be named: a remainder that is $o_p(1)$ is not necessarily negligible compared with $n^{-1/2}$. The probability limit $\theta_0$ need not be described as a true parameter for the algebra to apply; it is the value around which the estimator is being expanded.

Linearizing a Smooth Function

Suppose a vector of sample quantities $\hat{\eta}_n$ converges in probability to $\eta_0$, and $g$ is differentiable at $\eta_0$. If $\hat{\eta}_n-\eta_0=O_p(r_n)$, the first-order Taylor expansion gives

g(\hat{\eta}_n)-g(\eta_0) = Dg(\eta_0)(\hat{\eta}_n-\eta_0)+o_p(r_n).

Here $Dg(\eta_0)$ is the derivative matrix (or gradient in the scalar-output case), with dimensions that map changes in $\eta$ to changes in $g(\eta)$. The differentiability remainder is $o_p(\|\hat{\eta}_n-\eta_0\|)$; the assumed rate then makes it $o_p(r_n)$. If the second derivatives are bounded in a neighborhood of $\eta_0$, a Taylor remainder of order $O_p(r_n^2)$ is available. When $r_n\to0$, that bound implies $o_p(r_n)$.

These conclusions require the sample quantity to lie in the domain of $g$ with probability tending to one. For ratios, for example, the limiting denominator must be nonzero. For a square root, the limiting argument must be strictly positive. Such conditions ensure that the derivative is well-defined near the probability limit.

Worked Example: A Nonlinear Transformation of a Sample Mean

Let $Y_1,\ldots,Y_n$ be independent and identically distributed with $E[Y_i]=1$ and $\operatorname{Var}(Y_i)=0.25$. Define $\bar{Y}=n^{-1}\sum_{i=1}^n Y_i$, and, when $\bar{Y}>0.4$, set $\hat{\theta}_n=2\sqrt{\bar{Y}-0.4}$. (Define it arbitrarily otherwise.) Find its first-order expansion around its probability limit.

The law of large numbers gives $\bar{Y}\xrightarrow{p}1$, so $\hat{\theta}_n\xrightarrow{p}\theta_0=2\sqrt{0.6}$. The finite variance gives $\bar{Y}-1=O_p(n^{-1/2})$. Also, because $1>0.4$, the probability that $\bar{Y}>0.4$ tends to one. The function $g(y)=2\sqrt{y-0.4}$ is twice differentiable near $y=1$, with

g'(y)=(y-0.4)^{-1/2}, \qquad g'(1)=\frac{1}{\sqrt{0.6}}, \qquad g''(y)=-\frac{1}{2}(y-0.4)^{-3/2}.

A Taylor expansion therefore yields

\hat{\theta}_n-\theta_0 = \frac{1}{\sqrt{0.6}}(\bar{Y}-1) + O_p\!\left((\bar{Y}-1)^2\right) = \frac{1}{\sqrt{0.6}}(\bar{Y}-1)+O_p(n^{-1}).

Since $n^{-1}=o(n^{-1/2})$, the remainder is negligible at the $n^{-1/2}$ rate. Thus the expansion can also be written with $o_p(n^{-1/2})$ as its remainder. The coefficient $1/\sqrt{0.6}$ is exact; its numerical approximation is $1.29099$. Using that rounded value inside an expansion with an $o_p(n^{-1/2})$ remainder would generally be incorrect: the coefficient error times $\bar{Y}-1$ is of order $O_p(n^{-1/2})$, not $o_p(n^{-1/2})$.

For illustration, if a hypothetical sample has $n=100$ and $\bar{Y}=1.04$, then $\hat{\theta}_n=2\sqrt{0.64}=1.600$, while $\theta_0=2\sqrt{0.6}\approx1.54919$. The observed difference is approximately $0.05081$; its linear term is $(1/\sqrt{0.6})(0.04)\approx0.05164$. A finite-sample discrepancy is expected because the expansion also has a remainder.

Expansions From Estimating Equations

A second common route to an expansion is through an estimating equation. Let $\Psi_n(\theta)$ be a scalar sample function and suppose $\hat{\theta}_n$ satisfies $\Psi_n(\hat{\theta}_n)=0$. Write $\Delta_n=\hat{\theta}_n-\theta_0$. A useful sufficient set of conditions is that $\Delta_n=O_p(r_n)$, $\Psi_n(\theta_0)=O_p(r_n)$, and the derivative of $\Psi_n$ near $\theta_0$ converges in probability to a nonzero constant $A$. The derivative convergence should hold throughout the segment between $\theta_0$ and $\hat{\theta}_n$, with probability tending to one.

By the mean value theorem, for some point $\tilde{\theta}_n$ between $\theta_0$ and $\hat{\theta}_n$,

0=\Psi_n(\hat{\theta}_n) =\Psi_n(\theta_0)+\Psi_n'(\tilde{\theta}_n)\Delta_n.

Consistency places $\tilde{\theta}_n$ near $\theta_0$, and the derivative condition gives $\Psi_n'(\tilde{\theta}_n)\xrightarrow{p}A$. Since $A\ne0$, its reciprocal converges to $A^{-1}$. Solving and using $\Psi_n(\theta_0)=O_p(r_n)$ gives

\Delta_n = -\left[\Psi_n'(\tilde{\theta}_n)\right]^{-1}\Psi_n(\theta_0) = -A^{-1}\Psi_n(\theta_0)+o_p(r_n).

The last equality follows because the difference between the random reciprocal and $A^{-1}$ is $o_p(1)$, and multiplying it by an $O_p(r_n)$ quantity gives $o_p(r_n)$. This argument shows why a nonzero limiting derivative matters: if the derivative vanishes, this linear inversion is not justified and a different rate or expansion may be needed.

Worked Example: Expanding a Root of a Sample Equation

Suppose $X_1,\ldots,X_n$ are independent and identically distributed with $E[X_i]=2$ and finite variance. Define $\hat{\theta}_n$ as the positive solution to $\theta+\theta^2-\bar{X}=0$ when $\bar{X}>0$, and set $\hat{\theta}_n=0$ otherwise. Find the expansion around the positive population root.

The population equation is $\theta+\theta^2-2=0$, whose positive root is $\theta_0=1$. The sample root is

\hat{\theta}_n=\frac{-1+\sqrt{1+4\bar{X}}}{2}.

The law of large numbers implies $\bar{X}\xrightarrow{p}2$ and $\Pr(\bar{X}>0)\to1$, so the defined estimator converges to 1. The finite variance assumption gives $\bar{X}-2=O_p(n^{-1/2})$; the root is a differentiable function of $\bar{X}$ near 2, so $\hat{\theta}_n-1=O_p(n^{-1/2})$. On the event $\bar{X}>0$, direct substitution in the sample equation gives the exact identity, with $\Delta_n=\hat{\theta}_n-1$,

0=(1+\Delta_n)+(1+\Delta_n)^2-\bar{X} =-(\bar{X}-2)+3\Delta_n+\Delta_n^2.

Rearranging,

\Delta_n = \frac{1}{3}(\bar{X}-2)-\frac{1}{3}\Delta_n^2 = \frac{1}{3}(\bar{X}-2)+O_p(n^{-1}).

The last step uses $\Delta_n=O_p(n^{-1/2})$, so $\Delta_n^2=O_p(n^{-1})=o_p(n^{-1/2})$. The coefficient $1/3$ is the reciprocal of the derivative $1+2\theta_0=3$ of the population equation at the root.

For hypothetical data with $n=100$ and $\bar{X}=2.12$, the linear term is $(2.12-2)/3=0.0400$. The sample root is $(-1+\sqrt{9.48})/2\approx1.03948$, giving an actual difference of about $0.03948$. Their difference is about $-0.00052$, consistent with the quadratic remainder. The calculation illustrates the expansion; it does not assert a distribution for the estimator.

A Ratio Expansion and Its Conditions

For a ratio, the derivative must account for changes in both numerator and denominator. Let $(\bar{A},\bar{B})$ be sample means converging to $(\mu_A,\mu_B)$, where $\mu_B\ne0$, and suppose each sample mean differs from its limit by $O_p(n^{-1/2})$. The map $g(a,b)=a/b$ is differentiable near $(\mu_A,\mu_B)$. Its gradient there is $(1/\mu_B,-\mu_A/\mu_B^2)$, so the ratio has expansion

\frac{\bar{A}}{\bar{B}}-\frac{\mu_A}{\mu_B} = \frac{1}{\mu_B}(\bar{A}-\mu_A) - \frac{\mu_A}{\mu_B^2}(\bar{B}-\mu_B) + O_p(n^{-1}).

The denominator condition is substantive: because $\bar{B}\xrightarrow{p}\mu_B\ne0$, the sample denominator stays away from zero with probability tending to one. The second-order remainder is $O_p(n^{-1})$ when the second derivatives are bounded near the limit and the vector of sample-mean errors is $O_p(n^{-1/2})$.

Worked Example: Linearizing a Ratio of Sample Means

Suppose hypothetical paired observations have sample means $\bar{A}=2.10$ and $\bar{B}=3.90$, with probability limits $\mu_A=2$ and $\mu_B=4$. Assume the sample means converge at rate $n^{-1/2}$, and consider the ratio $\bar{A}/\bar{B}$.

The limiting ratio is $2/4=0.5$. The denominator limit is nonzero, and the derivatives at the limit are

\frac{\partial g}{\partial a}(2,4)=\frac{1}{4}, \qquad \frac{\partial g}{\partial b}(2,4)=-\frac{2}{16}=-\frac{1}{8}.

Thus the first-order change predicted by the linear term is

\frac{1}{4}(2.10-2)-\frac{1}{8}(3.90-4) = 0.025+0.0125 = 0.0375.

The exact sample ratio is $2.10/3.90\approx0.53846$, a difference of approximately $0.03846$ from 0.5. The gap between the exact difference and the linear term is about $0.00096$, attributable to the higher-order remainder. This one hypothetical pair of sample means illustrates the arithmetic; the stochastic order claim comes from the convergence-rate and smoothness conditions, not from those two numbers alone.

What the Expansion Does—and Does Not—Establish

A stochastic expansion matters because it reduces a complicated estimator to a leading term that can be studied using probability limits and convergence results. It also exposes which sample fluctuations affect the estimator to first order and which effects are smaller at the chosen rate. The expansion is therefore a bridge between an estimator’s construction and later inference, but it is not itself a confidence interval, a test, or a claim of asymptotic normality.

To use one responsibly, check three points: the estimator is close enough to the expansion point, the relevant derivative exists and is nonzero or otherwise invertible, and the remainder is negligible at the stated target rate. When the leading term is $O_p(n^{-1/2})$ and the remainder is $o_p(n^{-1/2})$, the remainder can be set aside for first-order analysis at that rate. It cannot automatically be discarded for a finer-order question.

Common Mistake: Treating a Small Remainder as Negligible at Any Rate

A frequent error is to see that a remainder tends to zero and conclude that it can be ignored in an $n^{-1/2}$-rate expansion. For example, $O_p(n^{-1/2})$ does tend to zero, but it is the same order as the leading term and is not generally negligible at that rate. The required statement is $o_p(n^{-1/2})$. Similarly, replacing an exact derivative coefficient with a rounded approximation can introduce an error of the same order as the leading term. Keep coefficients exact in the expansion, and give decimal approximations separately.

Key takeaway: A first-order stochastic expansion states an exact leading coefficient or matrix, a leading term with a specified rate, and a remainder that is negligible relative to that rate. Differentiability supplies the linear approximation; convergence rates and local regularity determine whether its remainder is small enough.

Check Your Understanding

For each question, identify the rate or condition that justifies the proposed expansion.

If $\hat{\eta}_n-\eta_0=O_p(n^{-1/2})$ and $g$ has bounded second derivatives near $\eta_0$, what order is the second-order Taylor remainder, and why is it negligible at rate $n^{-1/2}$?
Why does a ratio expansion require a nonzero probability limit for its denominator?
In a root expansion, what role does the limiting derivative $A\ne0$ play in obtaining a linear representation?
Is an $O_p(n^{-1/2})$ remainder negligible at the $n^{-1/2}$ rate? Explain the distinction from $o_p(n^{-1/2})$.
Why can a rounded derivative coefficient fail to preserve an expansion whose remainder is $o_p(n^{-1/2})$?