A Nonlinear Moment Estimator From Target to Limit Law

From a Conditional Mean to an Estimating Equation

Suppose observations $(Y_i,X_i)$, $i=1,\ldots,n$, are iid, and a scientifically specified model says that the conditional mean of the response is $m(X_i,\theta_0)$. The parameter $\theta_0$ is the target, and $m(x,\theta)$ may be nonlinear in the $p$-dimensional parameter $\theta$. Nonlinear least squares estimates $\theta_0$ by choosing the model curve whose predictions are closest to the observed responses in squared-error distance.

The key point is that the conditional-mean assumption supplies a population target, but it does not by itself give the estimator’s variance. The curvature of the model and the variability of the residuals both matter. Their roles will appear separately in the sandwich limit law.

Definition: Let $\Theta\subseteq\mathbb{R}^p$ be the parameter space, and suppose $m(x,\theta)$ is differentiable in $\theta$. Write $g(x,\theta)=\partial m(x,\theta)/\partial\theta$, a $p$-vector. The nonlinear least-squares estimator $\hat\theta$ minimizes $$ Q_n(\theta)=\frac{1}{n}\sum_{i=1}^n\{Y_i-m(X_i,\theta)\}^2 $$ over $\theta\in\Theta$. At an interior differentiable minimum it satisfies the estimating equation $$ \frac{1}{n}\sum_{i=1}^n g(X_i,\theta)\{Y_i-m(X_i,\theta)\}=0. $$

The equation follows by differentiating the objective: its gradient is minus twice the left side. The factor $-2$ does not affect its zero. In the notation of estimating equations, one observation contributes $\psi(Y,X,\theta)=g(X,\theta)\{Y-m(X,\theta)\}$. The residual is multiplied by the model’s parameter sensitivity: observations for which a small change in a parameter changes the fitted mean substantially receive correspondingly more influence on the equation.

Worked Example: Deriving and Solving a Nonlinear Equation

Let $X$ take values 0 and 1, and consider the model $m(x,\theta)=\theta+\theta^2x$, with $\theta\geq0$. For two observed pairs $(0,1)$ and $(1,2)$, find the nonlinear least-squares estimate.

The derivative is $g(x,\theta)=1+2\theta x$. Therefore the estimating equation is $$ (1)(1-\theta)+(1+2\theta)(2-\theta-\theta^2)=0. $$ Since $2-\theta-\theta^2=(1-\theta)(\theta+2)$, the left side factors as $$ (1-\theta)\{1+(1+2\theta)(\theta+2)\} =(1-\theta)(3+5\theta+2\theta^2). $$ For $\theta\geq0$, the second factor is strictly positive, so the only solution is $\hat\theta=1$. It also gives zero residual at both observations, hence $Q_2(1)=0$; since a sum of squares cannot be negative, this is a global minimum.

Why the Conditional Mean Identifies the Target

Write $\varepsilon=Y-m(X,\theta_0)$. The model assumption is $E[\varepsilon\mid X]=0$. For any candidate $\theta$, the residual can be decomposed into the true residual and a difference between fitted means. The cross term vanishes in expectation because that difference is a function of $X$.

Theorem (population target and consistency): Suppose the pairs $(Y_i,X_i)$ are iid, $E[Y^2]<\infty$, and $E[Y\mid X]=m(X,\theta_0)$. Let $\Theta$ be compact, and assume $m(X,\theta)$ is continuous in $\theta$ almost surely and $\sup_{\theta\in\Theta}|m(X,\theta)|$ has finite second moment. If $$ E[\{m(X,\theta)-m(X,\theta_0)\}^2]>0 \quad\text{for every }\theta\ne\theta_0, $$ then $\theta_0$ uniquely minimizes the population squared-error risk. If, in addition, the sample objectives converge uniformly in probability to that risk, every global minimizer $\hat\theta$ of $Q_n$ is consistent: $\hat\theta\xrightarrow{p}\theta_0$.

The stated integrability and continuity conditions, together with iid sampling and compactness, are standard sufficient conditions for a uniform law of large numbers for $Q_n$. More generally, uniform convergence may be assumed directly if another suitable set of conditions establishes it.

Proof. Define $Q(\theta)=E[\{Y-m(X,\theta)\}^2]$. Let $\Delta(X,\theta)=m(X,\theta)-m(X,\theta_0)$. Since $Y-m(X,\theta)=\varepsilon-\Delta(X,\theta)$, $$ Q(\theta)-Q(\theta_0) =E[\Delta(X,\theta)^2]-2E[\varepsilon\Delta(X,\theta)] =E[\Delta(X,\theta)^2]. $$ Indeed, iterated expectation gives $E[\varepsilon\Delta(X,\theta)]=E[\Delta(X,\theta)E[\varepsilon\mid X]]=0$. The identification condition makes this difference strictly positive whenever $\theta\ne\theta_0$, proving unique minimization.

To see why uniform convergence implies consistency, fix any $\epsilon>0$. Compactness and continuity make the set $C_\epsilon=\{\theta\in\Theta:\|\theta-\theta_0\|\geq\epsilon\}$ compact. If it is nonempty, continuity and unique minimization imply $$ \inf_{\theta\in C_\epsilon}Q(\theta)-Q(\theta_0)=\delta_\epsilon>0. $$ On the event $\sup_{\theta\in\Theta}|Q_n(\theta)-Q(\theta)|<\delta_\epsilon/3$, every $\theta\in C_\epsilon$ has $Q_n(\theta)>Q_n(\theta_0)$, so no global minimizer lies in $C_\epsilon$. The probability of the complementary event tends to zero by uniform convergence. Thus $P(\|\hat\theta-\theta_0\|\geq\epsilon)\to0$, as required. $\square$

Worked Example: Checking the Population Target

Let $X$ be equally likely to be 1 or 2, let $Y\mid X=x\sim N(x,1)$, and use $m(x,\theta)=\theta x$ with $\Theta=[0,2]$. The conditional mean is $X=m(X,1)$, so $\theta_0=1$. The population risk difference is $$ Q(\theta)-Q(1)=E[(\theta X-X)^2] =(\theta-1)^2 E[X^2] =\frac{5}{2}(\theta-1)^2. $$ Here $E[X^2]=(1^2+2^2)/2=5/2$, which is positive. Thus the unique population minimizer is 1. The compact parameter space, continuous model, and finite second moments meet the theorem’s sufficient conditions, so global minimizers of the sample objective are consistent.

The Sandwich Limit Distribution

For the limit law, define the two matrices $$ A=E[g(X,\theta_0)g(X,\theta_0)^{T}], \qquad B=E[g(X,\theta_0)g(X,\theta_0)^{T}\varepsilon^2], \quad \varepsilon=Y-m(X,\theta_0). $$ The first describes the expected local curvature of the estimating equation; the second is the covariance matrix of its contribution at the truth. Conditional heteroskedasticity generally makes $B$ different from a constant multiple of $A$.

Theorem (nonlinear least-squares sandwich limit): Suppose the consistency conditions above hold, $\theta_0$ is an interior point of $\Theta$, and $m(x,\theta)$ has continuous second derivatives in a neighborhood of $\theta_0$. Assume $A$ is finite and positive definite, $E[\|\psi(Y,X,\theta_0)\|^2]<\infty$, and the sample Jacobian of the estimating equation converges uniformly in probability in a neighborhood of $\theta_0$ to its population counterpart. Then a consistent interior solution of the estimating equation satisfies $$ \sqrt n(\hat\theta-\theta_0)\xrightarrow{d} N\!\left(0,A^{-1}BA^{-1}\right). $$

The Jacobian condition can, for example, be obtained from a uniform law of large numbers for the derivatives, using an integrable envelope and continuity near $\theta_0$. The assumed second moments give the multivariate central limit theorem for the estimating function.

Proof. At the true parameter, $E[\psi(Y,X,\theta_0)]=0$, since $E[\varepsilon\mid X]=0$. Differentiating one contribution gives $$ \frac{\partial\psi(Y,X,\theta)}{\partial\theta^T} = \frac{\partial^2m(X,\theta)}{\partial\theta\,\partial\theta^T} \{Y-m(X,\theta)\} -g(X,\theta)g(X,\theta)^T. $$ At $\theta_0$, the conditional mean of the first term given $X$ is zero. Hence the expected Jacobian is $-A$.

The estimating equation and the integral-Jacobian expansion about $\theta_0$ give $$ 0=\frac{1}{n}\sum_{i=1}^n\psi(Y_i,X_i,\theta_0) +\left\{\int_0^1\frac{1}{n}\sum_{i=1}^n \frac{\partial\psi(Y_i,X_i,\theta_0+t(\hat\theta-\theta_0))}{\partial\theta^T}\,dt\right\} (\hat\theta-\theta_0). $$ Consistency and the Jacobian condition make the integrated matrix in braces converge in probability to $-A$, whose inverse exists. Rearranging therefore yields $$ \sqrt n(\hat\theta-\theta_0) =A^{-1}\frac{1}{\sqrt n}\sum_{i=1}^n\psi(Y_i,X_i,\theta_0)+o_p(1). $$ The summands are iid, mean zero, and have covariance $B$. The multivariate central limit theorem and this expansion give the stated limit with covariance $A^{-1}BA^{-1}$. The matrix $A$ is symmetric, so its inverse is also symmetric. $\square$

Worked Example: A Nonlinear Transformation and Its Limit Variance

Suppose $Y_i$ are iid Poisson with mean 4 and use the conditional-mean model $m(\theta)=\theta^2/4$, with $\theta\geq0$. The target is $\theta_0=4$. Minimizing squared residuals gives $\hat\theta=2\sqrt{\bar Y}$, where $\bar Y=n^{-1}\sum_iY_i$.

Here $g(\theta)=\theta/2$, so $g(4)=2$. Since $\operatorname{Var}(Y_i)=4$, the scalar sandwich quantities are $A=2^2=4$ and $B=2^2\cdot4=16$. Thus the asymptotic variance of $\sqrt n(\hat\theta-4)$ is $B/A^2=16/16=1$.

The same result follows by differentiating the transformation $h(y)=2\sqrt y$: $h'(y)=1/\sqrt y$, so $h'(4)=1/2=0.5$. The delta-method variance is $(0.5)^2\cdot4=1$. In particular, the derivative is $0.5$, not $0.25$; using $0.25$ would incorrectly give an asymptotic variance of $0.25$.

Worked Example: Heteroskedasticity Changes the Sandwich

Let $X$ be equally likely to be 1 or 2, let $Y\mid X=1\sim N(1,1)$ and $Y\mid X=2\sim N(2,4)$, and fit $m(X,\theta)=\theta^2X$ with $\theta\geq0$. The true parameter is $\theta_0=1$. The derivative is $g(X,1)=2X$.

The curvature term is $$ A=E[(2X)^2]=\frac{4+16}{2}=10. $$ The score variance uses the conditional residual variances, which are 1 at $X=1$ and 4 at $X=2$: $$ B=E[(2X)^2\varepsilon^2] =\frac{4\cdot1+16\cdot4}{2} =34. $$ Consequently, the asymptotic variance of $\sqrt n(\hat\theta-1)$ is $B/A^2=34/100=0.34$. Replacing $B$ by $A$, or by an assumed common residual variance times $A$, would not reflect this model’s conditional variances.

Interpretation and a Common Pitfall

The sandwich form separates two sources of uncertainty: $A^{-1}$ translates perturbations in the estimating equation into parameter changes, while $B$ measures the variability of those perturbations. This is why the conditional-mean model can target the right parameter even when the conditional variance is not constant. Correctly specifying the mean does not imply that the residuals are homoskedastic or that a homoskedastic variance formula is valid.

Wrong reasoning: “The conditional mean is correct, so the residual variance must be constant and the estimator’s variance is just $A^{-1}$.”
Correction: The conditional-mean assumption makes the score mean zero and removes the expected residual-times-Hessian term from the Jacobian. It does not make $E[\varepsilon^2\mid X]$ constant. In general the covariance is $A^{-1}BA^{-1}$; the simplification to a multiple of $A^{-1}$ requires additional variance structure.
Another condition to check: A stationary solution of the estimating equation need not be a global least-squares minimum. Consistency concerns a suitably chosen global minimizer (or a consistent local solution), and asymptotic normality also requires local identification through nonsingular $A$.

Key takeaway: Nonlinear least squares converts a conditional-mean model into an estimating equation by multiplying each residual by the derivative of its fitted mean. Identification and uniform convergence justify consistency; a central limit theorem and a nonsingular Jacobian yield the sandwich limit law.

Check Your Understanding

Use the definitions and results above to answer these questions.

For $m(x,\theta)=\exp(\theta x)$, write the one-observation nonlinear least-squares estimating function.
Why does $E[Y\mid X]=m(X,\theta_0)$ make the cross term in the population risk difference equal to zero?
In the sandwich limit, what does $A$ represent, and what does $B$ represent?
Which part of the asymptotic variance can change when the conditional residual variance depends on $X$?
For the Poisson example, verify the derivative of $2\sqrt y$ at $y=4$ and the resulting delta-method variance.