Score Operators and Quadratic-Mean Differentiability

Likelihood Paths and Scores

In semiparametric analysis, a complicated model can be studied locally by examining one-dimensional paths through a reference distribution. A path describes how the distribution changes as a scalar parameter $t$ moves away from zero. Its derivative records the local direction of change. A score is the function of the data that represents this derivative on the likelihood scale.

For a parametric model with density $p_\theta(x)$, suppose $\theta$ is a vector and the path is $\theta(t)$, with $\theta(0)=\theta_0$ and derivative $\dot\theta(0)=a$. When the density is differentiable and the usual regularity conditions hold, differentiating its log density along the path gives $s(x)=a^\mathsf{T}\dot\ell_{\theta_0}(x)$, where $\dot\ell_{\theta_0}(x)$ is the vector of derivatives of $\log p_\theta(x)$ with respect to $\theta$ at $\theta_0$. Thus a path selects a directional score from the parametric score vector.

Definition: Let $P_0$ be a reference distribution and $P_t$ a one-dimensional path through it, with densities $p_t$ relative to a common dominating measure $\nu$. The path is quadratic-mean differentiable at $t=0$ with score $s$ if $s\in L^2(P_0)$ and $$ \int \left\{\sqrt{p_t(x)}-\sqrt{p_0(x)}-\frac{t}{2}s(x)\sqrt{p_0(x)}\right\}^2\,d\nu(x)=o(t^2) $$ as $t\to 0$. Equivalently, the square-root density has an $L^2(\nu)$ first-order expansion with derivative $\frac12s\sqrt{p_0}$. The score is defined up to equality $P_0$-almost surely.

The square-root scale is useful because densities have a fixed integral, while their square roots are unit vectors in $L^2(\nu)$. The factor $1/2$ makes the resulting score agree with the derivative of the log density whenever that pointwise derivative exists. Quadratic-mean differentiability (QMD) is stronger than merely differentiating a formula at each fixed observation: it controls the approximation after integrating over the whole sample space.

Why a Regular Score Has Mean Zero and Finite Variance

Finite variance is part of the QMD definition: $s\in L^2(P_0)$ means $E_0[s(X)^2]<\infty$. The zero-mean property follows from the fact that every density integrates to one. Here is the short derivation. Write $q_t=\sqrt{p_t}$, $q_0=\sqrt{p_0}$, and express the QMD expansion as $q_t=q_0+\frac{t}{2}s q_0+r_t$, where $\|r_t\|_{L^2(\nu)}=o(|t|)$. Since $\|q_t\|_2^2=\|q_0\|_2^2=1$, expanding the squared norm gives

0=t\int s(x)p_0(x)\,d\nu(x)+o(|t|).

To see the remainder order, the cross-product of $q_0$ and $r_t$ is $o(|t|)$ by the Cauchy–Schwarz inequality; the squared first-order term is $O(t^2)$, and all products involving $r_t$ are also $o(|t|)$. Divide by $t\ne0$ and let $t\to0$. The result is $E_0[s(X)]=0$. Thus the score is a centered, finite-variance random variable under the reference distribution.

Key result: Every QMD score satisfies $E_0[s(X)]=0$ and $E_0[s(X)^2]<\infty$. Its variance, $I=E_0[s(X)^2]$, is the information in that path at the reference distribution. For a vector score, the corresponding information matrix is the expected outer product of the score vector.

Worked Score Calculations

Worked Example: A Normal Location Path

Suppose hypothetical measurements $X$ follow $N(\mu,4)$, with known variance $4$, and consider the path $\mu_t=10+t$. The reference distribution is $P_0=N(10,4)$. Its density is smooth in $\mu$, has fixed support, and its square-root density is differentiable in $L^2$ (the normal density and its derivatives have integrable Gaussian tails). Thus this path is QMD.

The log density, apart from a constant, is $-\frac{(x-\mu_t)^2}{8}$. Differentiating with respect to $t$ at zero gives $s(x)=(x-10)/4$. Under $P_0$, $E_0[X-10]=0$ and $E_0[(X-10)^2]=4$, so

E_0[s(X)]=0,\qquad I=E_0[s(X)^2]=\frac{4}{16}=\frac14.

For hypothetical observed values $8,11,13$, the individual scores are $(8-10)/4=-0.5$, $(11-10)/4=0.25$, and $(13-10)/4=0.75$. Their sum is $0.50$. This observed sum is not required to equal zero: the zero-mean property is an expectation under $P_0$, not a restriction on every realized sample.

Worked Example: A Bernoulli Path and Its Information

Let $X$ be Bernoulli with success probability $p_t=0.4+0.1t$, for $t$ in a neighborhood of zero. The path stays inside $(0,1)$ for sufficiently small $|t|$, and the two outcome probabilities are smooth and positive there. Because the sample space has only two points, differentiating their square roots gives QMD directly.

For $x\in\{0,1\}$, the log likelihood is $x\log p_t+(1-x)\log(1-p_t)$. Its derivative at zero is

s(x)=0.1\left(\frac{x}{0.4}-\frac{1-x}{0.6}\right) =\frac{0.1(x-0.4)}{0.4(0.6)}.

The score is $0.25$ when $x=1$, and $-1/6$ when $x=0$. Therefore $E_0[s(X)]=0.4(0.25)+0.6(-1/6)=0.10-0.10=0$. Its variance is

I=0.4(0.25)^2+0.6(-1/6)^2 =0.025+\frac{1}{60} =\frac{1}{24}\approx0.04167.

Equivalently, using the Bernoulli variance formula gives $I=0.1^2/[0.4(0.6)]=0.01/0.24=1/24$. This exact value matters in later arithmetic: for example, $2^2I/2=4(1/24)/2=1/12\approx0.08333$, not $0.08334$ when calculated from the unrounded $I$.

Worked Example: A Poisson Rate Path

Suppose hypothetical counts have distribution $X\sim\operatorname{Poisson}(\lambda_t)$, where $\lambda_t=2+t$. For small $|t|$, the rate is positive and the support remains the nonnegative integers. The Poisson probabilities are smooth in the rate; the square-root probabilities and their derivatives are square-summable locally, so this path is QMD.

The log probability is $-\lambda_t+x\log\lambda_t-\log(x!)$. Differentiation at zero gives $s(x)=x/2-1$. Under $P_0$, the mean and variance of $X$ are both $2$, hence

E_0[s(X)]=\frac{E_0[X]}{2}-1=0,\qquad I=E_0\left[\left(\frac{X}{2}-1\right)^2\right] =\frac{\operatorname{Var}_0(X)}{4}=\frac12.

For hypothetical counts $2,0,3$, the scores are $0,-1,0.5$, with sum $-0.5$. Again, this sample score sum need not vanish; its expectation across samples under the reference rate is zero.

From QMD to a Local Likelihood Expansion

QMD also provides a rigorous route to a local expansion of the likelihood for independent observations. Let $X_1,\ldots,X_n$ be independent draws from $P_0$, and take a fixed real number $h$. Under the path distribution $P_{h/\sqrt n}$, the log likelihood ratio has the expansion

\log\frac{dP_{h/\sqrt n}^{\,n}}{dP_0^{\,n}} = h\Delta_n-\frac{h^2}{2}I+o_{P_0^n}(1), \qquad \Delta_n=\frac{1}{\sqrt n}\sum_{i=1}^n s(X_i), \quad I=E_0[s(X)^2].

This is the local asymptotic normality expansion for a one-dimensional QMD path. Its assumptions include independent sampling from the reference distribution, QMD at zero, and a fixed local shift $h$. The finite variance of the score gives $E_0[\Delta_n]=0$ and, by the central limit theorem, $\Delta_n$ converges in distribution to $N(0,I)$. The quadratic term is not an arbitrary correction: it is determined by the score variance and ensures the likelihood remains properly normalized to this order.

The expansion is in probability under $P_0^n$; it is not a claim that a finite-sample log likelihood ratio equals the displayed two leading terms exactly. Nor does QMD alone justify differentiating the log density pointwise at every observation. The controlled square-root-density remainder is what supports the asymptotic likelihood result.

Worked Example: The Bernoulli Local Likelihood at $n=100$

Continue with the Bernoulli path $p_t=0.4+0.1t$, for which $I=1/24$. Consider $n=100$ hypothetical independent observations with 44 successes. For $h=2$, the local path parameter is $t=h/\sqrt{n}=2/10=0.2$, so the alternative success probability is $p_t=0.4+0.1(0.2)=0.42$.

At the reference probability, the sum of the scores is $\sum_{i=1}^{100}s(X_i)=(5/12)(44-40)=5/3$, since $s(X)=(5/12)(X-0.4)$ for each observation. Therefore $\Delta_{100}=(5/3)/10=1/6$. The QMD local expansion gives

h\Delta_{100}-\frac{h^2I}{2} =2\left(\frac16\right)-\frac{4}{2}\left(\frac1{24}\right) =\frac13-\frac1{12} =\frac14=0.2500.

For comparison, the exact log likelihood ratio is $44\log(0.42/0.40)+56\log(0.58/0.60)\approx0.2483$. The closeness is consistent with the local expansion, while the small difference illustrates that the expansion has a remainder and is not an exact finite-sample identity.

Score Operators and Common Pitfalls

For a collection of differentiable paths through $P_0$, assigning to each path its score is often called the score operator. In a regular parametric model, this assignment agrees with directional differentiation: a path direction $a$ maps to $a^\mathsf{T}\dot\ell_{\theta_0}$. This language emphasizes that the score depends on the path, not just on a particular observed data set. Comparing and organizing sets of such scores is central to semiparametric analysis.

Pitfall: treating pointwise derivatives as sufficient. Differentiating $\log p_t(x)$ for each fixed $x$ does not by itself establish QMD. QMD requires the integrated squared remainder for square-root densities to be $o(t^2)$.
Pitfall: requiring an observed score to equal zero. The correct condition is $E_0[s(X)]=0$. A finite sample score sum can be positive or negative.
Pitfall: confusing score variance with observed curvature. The information $I=E_0[s^2]$ is a population quantity for the path. It is not generally the same as a particular sample’s squared score or a realized second derivative.
Pitfall: rounding too early. Keep exact quantities such as $I=1/24$ through calculations, then round the final result. This gives $2^2I/2=1/12\approx0.08333$.

A further caution concerns support. In the general QMD definition, densities are taken relative to a common dominating measure, and the expansion also controls probability appearing where $p_0=0$. A model whose support changes abruptly can fail QMD even if its density formula looks smooth on the interior of its support. Always check the distributional path, its support, and the integrated remainder—not just a formal derivative.

Key takeaway: A QMD path has a well-defined $L^2(P_0)$ score. Density normalization forces that score to have mean zero, QMD requires finite variance, and the same score and variance determine the first-order local likelihood term and its quadratic information correction.

Check Your Understanding

Answer each question using the definitions and calculations above.

State the QMD condition in terms of square-root densities and identify the order required of its $L^2$ remainder.
Why does normalization of $p_t$ imply $E_0[s(X)]=0$? Name the norm expansion used in the argument.
For the Bernoulli path $p_t=0.4+0.1t$, what are the scores at $X=1$ and $X=0$, and what is their expected value under $P_0$?
For the Poisson path $\lambda_t=2+t$, calculate the score at an observation $X=4$ and state the path information.
In the local likelihood expansion, what does $I$ represent, and why is the quadratic term not a finite-sample exact identity?

Score Operators and Quadratic-Mean Differentiability

What You'll Learn

Likelihood Paths and Scores

Why a Regular Score Has Mean Zero and Finite Variance

Worked Score Calculations

Worked Example: A Normal Location Path

Worked Example: A Bernoulli Path and Its Information

Worked Example: A Poisson Rate Path

From QMD to a Local Likelihood Expansion

Worked Example: The Bernoulli Local Likelihood at \(n=100\)

Score Operators and Common Pitfalls

Check Your Understanding

Score Operators and Quadratic-Mean Differentiability

What You'll Learn

Likelihood Paths and Scores

Why a Regular Score Has Mean Zero and Finite Variance

Worked Score Calculations

Worked Example: A Normal Location Path

Worked Example: A Bernoulli Path and Its Information

Worked Example: A Poisson Rate Path

From QMD to a Local Likelihood Expansion

Worked Example: The Bernoulli Local Likelihood at \(n=100\)

Score Operators and Common Pitfalls

Check Your Understanding

Previous

Next