Nuisance Tangent Spaces and Orthogonality

Why Nuisance Directions Matter

A statistical model can contain features of the distribution that matter for the target parameter and other features that must be estimated or accounted for but are not themselves the target. The latter are called nuisance features. To study local changes in a model, we describe them by score functions: a score records the first-order change in log density along a smooth one-dimensional path through the distribution.

The key distinction is whether a path changes the target. A path that varies the nuisance features while leaving the target fixed generates a nuisance score direction. Collecting these directions gives a geometric description of the ways the distribution can change without changing the target. Orthogonality to those directions can then be checked by an inner product under the distribution at which the analysis is being conducted.

Definition: Let $P_0$ be the distribution at which local changes are studied, and let $\theta(P)$ be the target parameter. A regular differentiable path $P_t$, with $P_{t=0}=P_0$, has score $s$ at $t=0$ if it satisfies the usual quadratic-mean differentiability condition and $s$ is its score function. The path is a nuisance path for $\theta$ if $\theta(P_t)=\theta(P_0)$ for all $t$ in a neighborhood of zero. Its score $s$ is a nuisance score direction.

Scores are elements of $L_2^0(P_0)$, the space of square-integrable functions with mean zero under $P_0$. The mean-zero property follows from differentiating the normalization of the density along a regular path. For instance, if $p_t$ is a density with respect to a common dominating measure, then $\int p_t=1$; differentiating at zero gives $\int s p_0=E_0[s]=0$. Here $E_0$ denotes expectation under $P_0$.

Definition: The nuisance tangent space, denoted $\mathcal{T}_{\mathrm{nuis}}$, is the closure in $L_2(P_0)$ of the linear span of all nuisance score directions generated by regular paths through $P_0$. A function $h\in L_2^0(P_0)$ is orthogonal to the nuisance tangent space if $$ \langle h,s\rangle_{P_0}=E_0[hs]=0 $$ for every $s\in\mathcal{T}_{\mathrm{nuis}}$.

The linear span allows combinations of directions, and the closure includes limits of such combinations in the $L_2(P_0)$ norm. Since inner products are continuous in that norm, it is enough to verify orthogonality to the generating nuisance scores. The inner product is weighted by $P_0$: in a discrete model, categories with higher probability contribute more.

A Discrete Model: Holding a Target Fixed

Consider a variable with three categories, labelled 1, 2, and 3. At $P_0$, let their probabilities be $1/4,\ 1/2,\ 1/4$. Define the target as the difference between the probabilities of categories 1 and 3, $\theta(P)=P(1)-P(3)$. A nuisance path can change the category probabilities while preserving this difference.

Worked Example: A Nuisance Score and Its Orthogonal Functions

For $-1<t<1$, define $$ P_t(1)=\frac14+\frac{t}{4},\qquad P_t(2)=\frac12-\frac{t}{2},\qquad P_t(3)=\frac14+\frac{t}{4}. $$ The probabilities are positive and sum to one on this range. Also, $\theta(P_t)=P_t(1)-P_t(3)=0=\theta(P_0)$, so the path holds the target fixed.

For category $i$, the score is the derivative of $\log P_t(i)$ at $t=0$, or equivalently $P'_0(i)/P_0(i)$. The derivatives of the three probabilities are $1/4,\ -1/2,\ 1/4$. Therefore the score values are $$ s(i)=\frac{P'_0(i)}{P_0(i)}=(1,-1,1). $$ The score is centered: $E_0[s]=(1/4)(1)+(1/2)(-1)+(1/4)(1)=0$.

Now take $h=(1,0,-1)$, with its entries corresponding to categories 1, 2, and 3. It is centered because $E_0[h]=(1/4)(1)+(1/2)(0)+(1/4)(-1)=0$. Its inner product with the nuisance score is $$ E_0[hs]=\frac14(1)(1)+\frac12(0)(-1)+\frac14(-1)(1)=0. $$ Thus $h$ is orthogonal to this score direction, and hence to the one-dimensional nuisance tangent space generated by the path.

In contrast, let $g=(-1,2,-1)$. Its mean is $E_0[g]=(1/4)(-1)+(1/2)(2)+(1/4)(-1)=1/2$, so $g$ is not centered. Moreover, $$ E_0[gs]=\frac14(-1)(1)+\frac12(2)(-1)+\frac14(-1)(1)=-\frac32. $$ It is therefore not orthogonal to the nuisance score. The centering failure and the nonzero inner product are separate facts; neither can be ignored when checking orthogonality.

This example illustrates both parts of the calculation. First, identify paths that preserve the target and calculate their scores. Then check an $L_2(P_0)$ inner product, using the probabilities under $P_0$ as weights. A plain unweighted dot product of the score values and function values would not be the relevant calculation.

Conditional Nuisance Variation

A common structure separates the distribution of covariates $X$ from the conditional distribution of an outcome $Y$ given $X$. Suppose the marginal distribution of $X$ is held fixed while the conditional law of $Y$ is unrestricted. Then nuisance paths vary the conditional distribution without changing the marginal law of $X$.

Characterization: In a model with fixed marginal distribution of $X$ and an unrestricted conditional distribution of $Y$ given $X$, conditional nuisance scores satisfy $E_0[s\mid X]=0$. In a sufficiently rich dominated model, the nuisance tangent space is the set of functions $s\in L_2^0(P_0)$ satisfying this conditional mean-zero restriction.

To see the restriction, factor the joint density as $p_X(x)p_t(y\mid x)$, with $p_X$ fixed along the path. The score comes only from the conditional density. For each $x$, the conditional density integrates to one, so differentiating that conditional normalization gives $E_0[s\mid X=x]=0$. Conversely, in an unrestricted conditional model, small conditional perturbations can generate square-integrable, conditionally centered scores (or their limits).

Worked Example: Testing Orthogonality in a Conditional Model

Let $X$ be binary with $P_0(X=0)=P_0(X=1)=1/2$. Suppose $Y\mid X=0$ is Bernoulli with probability $0.3$, while $Y\mid X=1$ is Bernoulli with probability $0.7$. Consider paths that keep the distribution of $X$ fixed and vary the two conditional Bernoulli probabilities.

For a path with conditional probability $p_x(t)=p_x+t b(x)$, the score is $$ s(X,Y)=b(X)\frac{Y-p_X}{p_X(1-p_X)}. $$ Indeed, differentiating the Bernoulli log likelihood $Y\log p_X(t)+(1-Y)\log(1-p_X(t))$ at zero gives this expression. Its conditional mean is zero because $E_0[Y-p_X\mid X]=0$. For the path with $b(x)=1$, the score takes values $10/3$ and $-10/7$ when $X=0$ and $Y=1$ and $Y=0$, respectively; when $X=1$, its values are $10/7$ and $-10/3$.

The function $h(X,Y)=2X-1$ is centered, since $X$ has equal probabilities at 0 and 1. For every conditional nuisance score, $$ E_0[hs]=E_0\!\left[h\,E_0[s\mid X]\right]=0. $$ Thus $h$, which depends only on $X$, is orthogonal to the conditional nuisance tangent space.

By comparison, $r(X,Y)=Y-1/2$ is centered because $E_0[Y]=(0.3+0.7)/2=1/2$, but it is not orthogonal to the score for $b(x)=1$. Conditional on either value of $X$, $E_0[(Y-1/2)s\mid X]=1$: the term involving the constant $p_X-1/2$ vanishes because $E_0[s\mid X]=0$, and $E_0[(Y-p_X)s\mid X]=1$. Hence $E_0[rs]=1$, not zero.

The conditional characterization also gives a useful general test. For a centered $h\in L_2(P_0)$, write $h=E_0[h\mid X]+\{h-E_0[h\mid X]\}$. If $h$ is a function of $X$, it is orthogonal to every conditionally centered score. Conversely, in the unrestricted conditional model, if $h$ is orthogonal to every nuisance score, its residual $q=h-E_0[h\mid X]$ is itself conditionally centered and belongs to the tangent space. Therefore $$ 0=E_0[hq]=E_0[q^2], $$ where the equality follows because $E_0[hq]=E_0[q^2]+E_0[E_0[h\mid X]q]$ and the second term is zero by conditional centering. Thus $q=0$ almost surely, so $h=E_0[h\mid X]$ almost surely. The restriction to a sufficiently rich conditional model matters: a smaller model may have fewer nuisance directions and a larger orthogonal complement.

A Parametric Illustration: Nuisance Scale

The same ideas apply when the nuisance variation has only a few dimensions. Suppose $Y\sim N(\mu,\sigma^2)$, the target is $\mu$, and $\sigma$ is nuisance. At $P_0$, let $\mu=0$ and $\sigma=1$. Consider paths that keep $\mu=0$ and vary $\sigma$.

Worked Example: Orthogonality to the Normal Scale Score

Take $\sigma_t=1+t$ for $t$ near zero. With $Z=Y$ at $P_0$, the log density along the path is $-\log(1+t)-Z^2/\{2(1+t)^2\}$ plus a constant. Differentiating at zero gives the nuisance score $$ s(Z)=Z^2-1. $$ It is centered because $E_0[Z^2]=1$.

The centered function $h(Z)=Z$ is orthogonal to this score: $$ E_0[hs]=E_0[Z(Z^2-1)]=E_0[Z^3]-E_0[Z]=0. $$ Both expectations vanish by symmetry of the standard normal distribution. Since this normal location-scale model has only this scale direction when $\mu$ is fixed, $Z$ is orthogonal to its nuisance tangent space.

The score itself is not orthogonal to the nuisance tangent space: $E_0[s^2]=E_0[(Z^2-1)^2]=3-2+1=2$, using $E_0[Z^4]=3$. This positive inner product confirms that $s$ is a nonzero nuisance direction.

How to Check Orthogonality

A reliable calculation follows a short sequence. The path restriction comes first: without verifying that the target is fixed, its score need not be a nuisance direction. The inner product comes second: it must use the distribution $P_0$ and, for discrete outcomes, the correct probability weights.

Specify the target and base distribution.
Record the parameter being held fixed and the distribution $P_0$ that determines expectations and inner products.

Identify admissible paths.
Check that each path is regular and that its target value stays equal to $\theta(P_0)$ near zero.

Calculate and center the scores.
Differentiate the log density along each path, and verify that the resulting score has mean zero.

Form the tangent space and test.
Take the closed linear span of the nuisance scores, then evaluate $E_0[hs]$ for its generators.

Common Mistakes and Why the Geometry Matters

Using a direction that changes the target. A score is a nuisance direction only if it comes from a path that holds the target fixed. A mean-zero score alone is not enough.
Checking an unweighted dot product. The inner product is $E_0[hs]$, so discrete values must be weighted by their probabilities under $P_0$.
Forgetting to check centering. The usual tangent space is a subspace of $L_2^0(P_0)$. A candidate function intended to lie in that space must have mean zero. As the discrete example shows, a noncentered function may also fail the orthogonality calculation itself.
Checking only one score when there are several directions. Orthogonality to a tangent space requires orthogonality to every generating nuisance score, not just to one convenient path. In a finite-dimensional model, this means checking a spanning set.
Assuming the conditional result holds in a restricted model. The condition $E_0[s\mid X]=0$ describes a rich model allowing unrestricted conditional variation with the marginal law of $X$ fixed. Additional restrictions can shrink the tangent space.

Nuisance tangent spaces make local model variation explicit: they collect the score directions available when the target is held fixed. Orthogonality then means that a centered function has zero $P_0$-weighted inner product with every such direction. This is a geometric statement about the model at $P_0$; it is not, by itself, a claim that an estimator is robust or efficient.

Check Your Understanding

Use the definitions and calculations above to answer these questions.

In the three-category example, why is $(1,-1,1)$ a nuisance score direction even though the category probabilities change along its path?
For probabilities $(1/4,1/2,1/4)$, calculate the mean of $g=(-1,2,-1)$. Is it centered, and what is its inner product with the score $(1,-1,1)$?
In a model with fixed marginal distribution of $X$ and unrestricted conditional variation, what conditional-mean restriction must a nuisance score satisfy?
Why is a centered function of $X$ orthogonal to every conditionally centered nuisance score?
For the standard normal scale score $Z^2-1$, calculate its inner product with $Z$. Which symmetry-based moments determine the result?