Orthogonal Scores and Nuisance-Estimation Error

Why Orthogonality Matters

Many econometric estimators target a parameter of interest while also estimating nuisance functions or parameters. A plug-in estimator may be consistent when those nuisance estimates are consistent, yet their errors can still affect the target at first order. That matters for inference: a first-order nuisance effect can change the target estimator’s limiting distribution or make a conventional standard error invalid.

An orthogonal score is constructed so that small changes in the nuisance argument have no first-order effect on its population moment at the truth. This removes one source of first-order error; it does not make nuisance estimation irrelevant. The score must still be sufficiently smooth, nuisance estimates must converge quickly enough for the remaining error to vanish at the target’s sampling scale, and the empirical score must obey an appropriate central limit theorem.

Definition: Let $W$ denote an observation, $\theta_0$ a target parameter, and $\eta_0$ a nuisance parameter or function. A score $\psi(W;\theta,\eta)$ is Neyman-orthogonal with respect to $\eta$ at $(\theta_0,\eta_0)$ if the derivative of its population moment in every admissible nuisance direction $h$ is zero: $$ \left.\frac{d}{dr}E\!\left[\psi(W;\theta_0,\eta_0+rh)\right]\right|_{r=0}=0. $$ The target moment is assumed to satisfy $E[\psi(W;\theta_0,\eta_0)]=0$. For a finite-dimensional nuisance parameter, orthogonality means that the gradient of this population moment with respect to $\eta$ is zero at $\eta_0$.

How the Nuisance Error Enters

For clarity, consider a scalar target $\theta_0$ and a scalar score. Write $P f=E[f(W)]$ and $P_n f=n^{-1}\sum_{i=1}^n f(W_i)$. Suppose the estimator $\hat{\theta}$ approximately solves $P_n\psi(W;\hat{\theta},\hat{\eta})=0$. The key distinction is between the population moment $P\psi$, which describes systematic changes in the score, and the sample fluctuation $(P_n-P)\psi$.

P_n\psi(\theta_0,\hat{\eta}) = (P_n-P)\psi(\theta_0,\eta_0) + P\{\psi(\theta_0,\hat{\eta})-\psi(\theta_0,\eta_0)\} + (P_n-P)\{\psi(\theta_0,\hat{\eta})-\psi(\theta_0,\eta_0)\}.

The first term is the sampling fluctuation of the score at the truth. Orthogonality controls the second, population term. The third is the empirical-process contribution from using an estimated nuisance argument in the sample score. Orthogonality alone does not set that third term to zero.

Suppose the population moment has a second-order expansion in the nuisance error, with a remainder bounded in magnitude by a constant times $r_n^2$, where $\|\hat{\eta}-\eta_0\|=O_p(r_n)$. The linear term vanishes by orthogonality, so the population nuisance contribution is $O_p(r_n^2)$. To make it negligible compared with the target’s usual $n^{-1/2}$ sampling error, a sufficient rate condition is $\sqrt{n}\,r_n^2\to 0$. This is often called a faster-than-$n^{-1/4}$ nuisance rate requirement, in the sense that $r_n=o(n^{-1/4})$.

Worked Example: An Exactly Quadratic Nuisance Remainder

Consider the hypothetical score $\psi(Y;\theta,\eta)=Y-\theta+\eta^2$, with nuisance truth $\eta_0=0$ and target $\theta_0=E[Y]$. At the truth, $E[\psi(Y;\theta_0,0)]=0$. For any direction $h$, the derivative of the population moment in that direction is $$ \left.\frac{d}{dr}E[\psi(Y;\theta_0,rh)]\right|_{r=0} = \left.\frac{d}{dr}r^2h^2\right|_{r=0}=0. $$ Thus the score is orthogonal at $\eta_0=0$, and its population nuisance error is exactly $(\hat{\eta}-\eta_0)^2$.

Suppose a hypothetical sample has $Y$ values 8, 10, 11, and 13. Then $\bar{Y}=(8+10+11+13)/4=10.5$. If $\hat{\eta}=0.2$, solving the sample score equation gives $$ 0=\bar{Y}-\hat{\theta}+\hat{\eta}^2 \quad\Longrightarrow\quad \hat{\theta}=10.5+(0.2)^2=10.54. $$ The nuisance estimate shifts the target estimate by $0.04$, not by an amount linear in $0.2$. This finite-sample calculation demonstrates the second-order remainder; by itself, it does not establish that the shift is negligible as $n$ grows.

A Partially Linear Score

A useful illustration is the partially linear regression model. Let $D$ be a scalar regressor, $X$ a vector of covariates, and $Y$ an outcome. Define $m_0(X)=E[D\mid X]$ and $l_0(X)=E[Y\mid X]$. Suppose the conditional mean of $Y$ satisfies $E[Y\mid D,X]=l_0(X)+\theta_0\{D-m_0(X)\}$, with $E[(D-m_0(X))^2]>0$. A score for $\theta_0$ is $$ \psi(W;\theta,\eta) = \{D-m(X)\}\{Y-l(X)-\theta(D-m(X))\}, \qquad \eta=(l,m). $$ At the truth, $E[Y-l_0(X)\mid X]=0$ and $E[D-m_0(X)\mid X]=0$. The model implies $E[\psi(W;\theta_0,\eta_0)]=0$.

Worked Example: Checking Orthogonality in the Partially Linear Score

Consider hypothetical population moments with two equally likely covariate groups. In each group, let the conditional means be $m_0(X)=1$ or 3 and $l_0(X)=2$ or 5, respectively. Suppose within each group $E[(D-m_0(X))^2\mid X]=1$, and $E[(D-m_0(X))(Y-l_0(X))\mid X]=2$. The score’s population moment at $\theta_0=2$ is therefore $2-2(1)=0$.

Now perturb $l_0$ in a direction $a(X)$. The directional derivative of the population score is $-E[(D-m_0(X))a(X)]$. Conditional on $X$, the residual $D-m_0(X)$ has mean zero, so this derivative is zero.

Next perturb $m_0$ in a direction $b(X)$. Differentiating both appearances of $m$ gives the derivative $$ E\!\left[-b(X)\{Y-l_0(X)-\theta_0(D-m_0(X))\} +\theta_0 b(X)\{D-m_0(X)\}\right]. $$ The conditional mean of $Y-l_0(X)$ is zero, as is that of $D-m_0(X)$. Taking conditional expectations given $X$ therefore makes both contributions zero. The score is orthogonal in both nuisance directions. The check relies on the stated conditional-mean restrictions, not merely on a convenient algebraic form.

Conditions for Target-Parameter Inference

Orthogonality supplies only one part of an asymptotic argument. The following conditions give a standard sufficient route to root-$n$ inference for a scalar target:

The observations are independent and identically distributed, or satisfy conditions that deliver the same relevant law of large numbers and central limit theorem.
The true score has mean zero and finite, positive variance: $E[\psi(W;\theta_0,\eta_0)]=0$ and $0<\Omega=E[\psi(W;\theta_0,\eta_0)^2]<\infty$.
The population moment is differentiable in $\theta$, with $A=\partial_\theta E[\psi(W;\theta_0,\eta_0)]\ne 0$, and the estimator is consistent and solves the sample equation up to an error negligible at the $n^{-1/2}$ scale.
The nuisance estimate is consistent, the orthogonal population remainder is $o_p(n^{-1/2})$, and the empirical-process contribution $(P_n-P)\{\psi(\theta_0,\hat{\eta})-\psi(\theta_0,\eta_0)\}$ is $o_p(n^{-1/2})$.
The score and moment are regular enough for the local expansion in $\theta$ and the nuisance remainder bound to hold.

Under these conditions, expanding the sample moment in $\theta$ around $\theta_0$ yields the asymptotic linear representation below. For an approximate root, its equation-solving error must also be $o_p(n^{-1/2})$.

\sqrt{n}(\hat{\theta}-\theta_0) = -A^{-1}\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\psi(W_i;\theta_0,\eta_0)+o_p(1) \ \xrightarrow{d}\ N\!\left(0,\frac{\Omega}{A^2}\right).

The sign in the representation follows from the first-order expansion $0\approx P_n\psi(\theta_0,\hat{\eta})+A(\hat{\theta}-\theta_0)$. The central limit theorem applies to the sum of scores at the truth; the nuisance terms have been required to be smaller than its $n^{-1/2}$ scale. A consistent estimate of $\Omega/A^2$ then gives a standard error for $\hat{\theta}$, and a large-sample confidence interval can be formed using the normal approximation. For a vector target, $A$ is replaced by the Jacobian matrix of the population moment, which must be nonsingular.

Worked Example: Comparing Nuisance Rates

Suppose a hypothetical estimator has a second-order population remainder bounded by a constant times $r_n^2$, and its empirical-process contribution is separately known to be $o_p(n^{-1/2})$. If the nuisance rate is $r_n=n^{-1/3}$, then $$ \sqrt{n}\,r_n^2=n^{1/2}n^{-2/3}=n^{-1/6}\longrightarrow 0. $$ The population nuisance remainder is therefore negligible at the root-$n$ scale, subject to the other inference conditions.

By contrast, if $r_n=n^{-1/4}$, then $\sqrt{n}\,r_n^2=1$: the second-order remainder need not vanish at that scale. A rate of exactly $n^{-1/4}$ is not enough under this generic bound; a faster rate is needed, unless some additional structure makes the actual remainder smaller.

For a concrete scale comparison at $n=2500$, the bound $r_n=n^{-1/3}$ gives $\sqrt{n}r_n^2=2500^{-1/6}\approx 0.2714$, while $r_n=n^{-1/4}$ gives $\sqrt{n}r_n^2=1$. These finite-$n$ values illustrate the rate comparison; they do not establish the required asymptotic conditions or determine a confidence interval.

Why Empirical Error Still Needs Attention

The population derivative condition concerns $P\psi$. In a sample, $\hat{\eta}$ is itself random, and its error can interact with the observations used to evaluate the score. One common sufficient strategy is to estimate nuisance functions on one part of the data and evaluate the score on observations not used for that nuisance fit. With suitable conditional independence across these parts and a vanishing $L^2$ difference between the fitted and true scores, the centered empirical-process term can be controlled. The details depend on the score and the sampling design; sample splitting does not remove the need for the population rate condition.

If the same observations are used to fit flexible nuisance functions and evaluate the score, the empirical-process term may require stronger complexity or stochastic-equicontinuity conditions. It should not simply be discarded because the population score is orthogonal. In either approach, the relevant goal is to show that the empirical nuisance contribution is $o_p(n^{-1/2})$.

Common Mistake

Wrong reasoning: “The score is orthogonal, so nuisance-estimation error has no effect on target inference.”

Correction: Orthogonality removes the first derivative of the population moment with respect to nuisance perturbations at the truth. Under suitable smoothness, it leaves a second-order population remainder. That remainder must still be negligible at the target’s sampling scale, and the empirical-process contribution must also be controlled. A slow nuisance rate, a large remainder, or uncontrolled dependence between nuisance fitting and score evaluation can invalidate the usual root-$n$ approximation.

Key takeaway: Orthogonality changes the leading nuisance error from first order to second order; it does not guarantee valid inference on its own. Establish the derivative condition, bound the remaining population error, control the sample fluctuation, and verify the target moment’s regularity before using a root-$n$ standard error.

Check Your Understanding

Use the definitions and conditions above to answer these questions.

State the directional-derivative condition that defines orthogonality with respect to a nuisance function.
If the population nuisance remainder is $O_p(r_n^2)$, what rate condition makes it $o_p(n^{-1/2})$?
Why does a zero population derivative not, by itself, control the empirical-process contribution?
In the partially linear score, which conditional mean-zero restrictions make the derivative in $l(X)$ and $m(X)$ directions vanish?
What additional role do $A\ne 0$ and finite positive $\Omega$ play in the root-$n$ inference result?