Gaussian Risk Expansion for X Plus g(X)

Why the Cross Term Matters

Suppose a Gaussian observation $X$ is used directly to estimate its mean $\mu$, but we modify that estimate by adding a function of the observation. The resulting rule is $\delta(X)=X+g(X)$. Its error contains both the original observation error and the adjustment. Squaring that error creates a cross term: it records whether the adjustment tends to align with or oppose the noise in $X$.

Although $E[X-\mu]=0$, the cross term is not generally zero. The adjustment $g(X)$ depends on the same random vector as the noise, so the two can be associated. Stein’s identity converts this dependence into an expectation of derivatives. This tutorial derives that conversion and the resulting risk expansion.

Definition: Let $X\in\mathbb{R}^p$ have distribution $N_p(\mu,\sigma^2 I_p)$, where $\sigma^2>0$, and let $g:\mathbb{R}^p\to\mathbb{R}^p$ be a measurable adjustment. The estimator is $\delta(X)=X+g(X)$. Under squared-error loss, its risk at $\mu$ is $$ R(\mu,\delta)=E_\mu\!\left[\|\delta(X)-\mu\|^2\right], $$ provided this expectation is finite. When the coordinate derivatives exist in the appropriate sense, define $\operatorname{div}g(x)=\sum_{i=1}^p \partial_i g_i(x)$.

Here $\|v\|^2=\sum_{i=1}^p v_i^2$, and $g_i$ is the $i$th coordinate of $g$. The divergence is the sum of the derivatives of each adjustment coordinate with respect to its matching observation coordinate. It is not the sum of all entries of the derivative matrix.

Evaluating the Cross Term

The needed form of Stein’s identity applies coordinate by coordinate. The following conditions are sufficient and make explicit where integration by parts is used. For each $i$, assume $g_i$ has a weak partial derivative $\partial_i g_i$, and that for almost every fixed value of the other coordinates, its section as a function of $x_i$ is locally absolutely continuous. Assume also that $E_\mu[g_i(X)^2]$ and $E_\mu[|\partial_i g_i(X)|]$ are finite, and that the boundary values of $g_i(x)$ times the one-dimensional Gaussian density in coordinate $i$ tend to zero as $x_i\to\pm\infty$, for almost every such section. These assumptions permit one-dimensional integration by parts and integration over the remaining coordinates.

Lemma: Under the stated regularity conditions, for each $i=1,\ldots,p$, $$ E_\mu\!\left[(X_i-\mu_i)g_i(X)\right] =\sigma^2 E_\mu\!\left[\partial_i g_i(X)\right]. $$ Consequently, $$ E_\mu\!\left[(X-\mu)^\mathsf{T}g(X)\right] =\sigma^2 E_\mu\!\left[\operatorname{div}g(X)\right]. $$

Proof. Fix the values $x_{-i}$ of all coordinates other than $x_i$. Write $\phi_i(t)$ for the $N(\mu_i,\sigma^2)$ density. Its derivative is $$ \phi_i'(t)=-\frac{t-\mu_i}{\sigma^2}\phi_i(t). $$ For almost every fixed $x_{-i}$, integration by parts on a finite interval and then passage to the infinite endpoints gives $$ \int_{\mathbb{R}}(t-\mu_i)g_i(t,x_{-i})\phi_i(t)\,dt =-\sigma^2\int_{\mathbb{R}}g_i(t,x_{-i})\phi_i'(t)\,dt =\sigma^2\int_{\mathbb{R}}\partial_i g_i(t,x_{-i})\phi_i(t)\,dt. $$ The boundary term vanishes by assumption. The integrals can then be integrated over $x_{-i}$: the derivative term is absolutely integrable by assumption, and the left-hand term is integrable by Cauchy–Schwarz, since $$ E_\mu\!\left[|(X_i-\mu_i)g_i(X)|\right] \leq \sqrt{E_\mu[(X_i-\mu_i)^2]}\sqrt{E_\mu[g_i(X)^2]}<\infty. $$ The joint Gaussian density factors into the density in coordinate $i$ and the density of the remaining coordinates, so this integration yields the first identity. Summing it over $i$ gives the second identity by linearity of expectation and the definition of divergence. $\square$

The Gaussian Risk Expansion

We can now separate the risk into the risk of $X$ itself, the squared size of the adjustment, and the cross term. The regularity conditions above are assumed, together with $E_\mu[\|g(X)\|^2]<\infty$, so all terms in the expansion are finite.

Theorem: If $X\sim N_p(\mu,\sigma^2 I_p)$, $\delta(X)=X+g(X)$, and the conditions for the lemma hold, then $$ R(\mu,\delta) =p\sigma^2 +E_\mu[\|g(X)\|^2] +2\sigma^2 E_\mu[\operatorname{div}g(X)]. $$

Proof. Since $\delta(X)-\mu=(X-\mu)+g(X)$, the squared Euclidean norm expands as $$ \|\delta(X)-\mu\|^2 =\|X-\mu\|^2 +2(X-\mu)^\mathsf{T}g(X) +\|g(X)\|^2. $$ Take expectations. The coordinates of $X-\mu$ each have variance $\sigma^2$, so $$ E_\mu[\|X-\mu\|^2] =\sum_{i=1}^p E_\mu[(X_i-\mu_i)^2] =p\sigma^2. $$ By the lemma, the expected inner product is $\sigma^2 E_\mu[\operatorname{div}g(X)]$. Substituting these two expectations into the expansion gives $$ R(\mu,\delta) =p\sigma^2 +2\sigma^2 E_\mu[\operatorname{div}g(X)] +E_\mu[\|g(X)\|^2], $$ as claimed. $\square$

The coefficient $2$ in front of the divergence expectation comes from the algebraic cross term, while the factor $\sigma^2$ comes from Stein’s identity. The adjustment’s squared norm contributes with a positive sign. The divergence term can have either sign, so it can offset some or all of that contribution.

Worked Calculations

Worked Example: A Constant Adjustment

Let $p=2$, $\sigma^2=3$, and $g(x)=a$, where $a=(1,-2)^\mathsf{T}$. The estimator adds the same vector regardless of the observation. Its divergence is zero because each coordinate is constant, and $\|a\|^2=1^2+(-2)^2=5$. The expansion gives $$ R(\mu,\delta)=2(3)+5+2(3)(0)=11. $$

This can be checked directly. The estimation error is $X-\mu+a$, so $$ E_\mu[\|X-\mu+a\|^2] =E_\mu[\|X-\mu\|^2]+2a^\mathsf{T}E_\mu[X-\mu]+\|a\|^2 =6+0+5=11. $$ The cross term vanishes here because the adjustment is constant, not merely because $X-\mu$ has mean zero.

Worked Example: A Linear Adjustment in Two Dimensions

Let $X\sim N_2(\mu,4I_2)$, with $\mu=(3,-1)^\mathsf{T}$, and take $g(x)=-x/4$. Then $\delta(X)=3X/4$. The divergence is $-1/4-1/4=-1/2$, and $$ E_\mu[\|g(X)\|^2] =\frac{1}{16}E_\mu[\|X\|^2] =\frac{1}{16}\left(\|\mu\|^2+2\sigma^2\right) =\frac{1}{16}(10+8)=\frac{9}{8}. $$

The risk expansion therefore gives $$ R(\mu,\delta) =2(4)+\frac{9}{8}+2(4)\left(-\frac{1}{2}\right) =8+\frac{9}{8}-4 =\frac{41}{8}. $$

For a direct check, write $\delta(X)-\mu=\frac{3}{4}(X-\mu)-\frac{1}{4}\mu$. The random centered term has mean zero, so its cross product with the constant term has expectation zero. Thus $$ R(\mu,\delta) =\frac{9}{16}E_\mu[\|X-\mu\|^2]+\frac{1}{16}\|\mu\|^2 =\frac{9}{16}(8)+\frac{10}{16} =\frac{72+10}{16} =\frac{41}{8}. $$

Worked Example: A Nonlinear Adjustment in One Dimension

Let $X\sim N(1,1)$ and $g(x)=x^2/4$. Then $\delta(X)=X+X^2/4$, and $g'(x)=x/2$, so $$ E[g'(X)]=\frac{E[X]}{2}=\frac{1}{2}. $$ Also, $E[g(X)^2]=E[X^4]/16$. For a normal variable with mean $1$ and variance $1$, $$ E[X^4]=1^4+6(1^2)(1)+3(1^2)=10, \qquad E[g(X)^2]=\frac{10}{16}=\frac{5}{8}. $$ The risk expansion gives $$ R(1,\delta)=1+\frac{5}{8}+2(1)\left(\frac{1}{2}\right)=\frac{21}{8}. $$

To verify the cross term directly, expand the error as $(X-1)+X^2/4$. Its cross contribution to the squared error is $$ 2E\!\left[(X-1)\frac{X^2}{4}\right] =\frac{1}{2}\bigl(E[X^3]-E[X^2]\bigr) =\frac{1}{2}(4-2)=1, $$ where $E[X^2]=2$ and $E[X^3]=1^3+3(1)(1)=4$. The other terms are $E[(X-1)^2]=1$ and $E[X^4]/16=5/8$, giving $1+1+5/8=21/8$, in agreement with the expansion.

Common Pitfall and Interpretation

A common incorrect argument is: “$E[X-\mu]=0$, so $E[(X-\mu)^\mathsf{T}g(X)]=0$.” This would be valid if $g(X)$ were a fixed vector independent of the centered noise, but $g(X)$ is generally a function of that noise. For example, in the nonlinear calculation above, the cross contribution is $1$, not $0$. Stein’s identity accounts for this dependence through the expected derivative.

The expansion is an exact identity, not by itself a guarantee that a particular adjustment improves risk. The squared-adjustment term is nonnegative, whereas the expected divergence term may be negative or positive. Their combined contribution determines how the risk differs from the baseline $p\sigma^2$. Applying the identity also requires regularity: a formal derivative calculation is not enough if the boundary or integrability conditions needed for integration by parts fail.

Key takeaway: Expand the squared error before simplifying. The cross term is $2E_\mu[(X-\mu)^\mathsf{T}g(X)]$, and Stein’s identity evaluates it as $2\sigma^2E_\mu[\operatorname{div}g(X)]$ when its regularity conditions hold.

Check Your Understanding

Use the expansion and its conditions to answer the following.

For $X\sim N_p(\mu,\sigma^2I_p)$, what is the risk of the unadjusted estimator $\delta(X)=X$?
Why does $E_\mu[X-\mu]=0$ not imply that the cross term vanishes for a general $g(X)$?
If $g(x)=a$ is constant, what are $E_\mu[\|g(X)\|^2]$ and $\operatorname{div}g(x)$?
For a differentiable vector field, which coordinate derivatives are summed to form $\operatorname{div}g$?
Which regularity features are needed to justify the coordinatewise integration by parts in the lemma?