Divergence Formula for Risk Difference

From a Risk Expansion to a Divergence Formula

Suppose $X$ is a noisy observation of a vector mean $\mu$, and consider an estimator that adjusts the observation by a function of the data. The adjustment can help by counteracting noise, but it also has a direct squared cost. The divergence formula makes these competing effects explicit: it expresses the risk change as the expected sum of a nonnegative adjustment cost and a term measuring how the adjustment changes across the sample space.

We use squared-error loss and the isotropic Gaussian model. Write $X\sim N_p(\mu,\sigma^2 I_p)$, where $I_p$ is the $p$-dimensional identity matrix and $\sigma^2>0$ is known. The estimator is $\delta(X)=X+g(X)$, with $g:\mathbb R^p\to\mathbb R^p$. Its risk at $\mu$ is the expected squared distance from the estimator to the true mean.

Definition: For an estimator $\delta$, its quadratic risk at $\mu$ is $R(\mu,\delta)=E_\mu[\|\delta(X)-\mu\|^2]$, where $\|v\|^2=\sum_{i=1}^p v_i^2$. For a differentiable vector field $g=(g_1,\ldots,g_p)$, its divergence is $\operatorname{div}g(x)=\sum_{i=1}^p \partial g_i(x)/\partial x_i$. The estimator $X$ itself has risk $R(\mu,X)=p\sigma^2$.

The divergence sums only the partial derivative of each adjustment coordinate with respect to its corresponding data coordinate. It is not the sum of all derivatives of all components. The result below compares the adjusted estimator with $X$, under the same model and loss.

The Risk Difference Theorem

The Gaussian risk expansion for $X+g(X)$ separates the baseline risk, the squared adjustment, and a cross term involving $X-\mu$. Stein’s identity converts that cross term into an expected divergence. For clarity, the regularity conditions needed for that conversion are stated here.

Regularity conditions: Assume $E_\mu[\|g(X)\|^2]<\infty$. For each coordinate $i$, assume $g_i$ is continuously differentiable in $x_i$ for fixed values of the other coordinates, $E_\mu[|\partial g_i(X)/\partial x_i|]<\infty$, and $E_\mu[|(X_i-\mu_i)g_i(X)|]<\infty$. Also assume that for almost every fixed value of the other coordinates, $g_i(x)$ times the $N(\mu_i,\sigma^2)$ density tends to zero at both ends of the $x_i$-axis, and that the integrals permit Fubini’s theorem. These conditions ensure the coordinatewise Gaussian integration-by-parts identity applies.

Theorem: Under the model and regularity conditions above, $$ R(\mu,\delta)-R(\mu,X) = E_\mu\!\left[\|g(X)\|^2+2\sigma^2\operatorname{div}g(X)\right]. $$

Proof. By definition, $\delta(X)-\mu=(X-\mu)+g(X)$. Expanding the squared norm and taking expectations gives

R(\mu,\delta) = E_\mu[\|X-\mu\|^2] +E_\mu[\|g(X)\|^2] +2E_\mu[(X-\mu)^\mathsf{T}g(X)].

Since the coordinates of $X-\mu$ each have variance $\sigma^2$, $E_\mu[\|X-\mu\|^2]=p\sigma^2=R(\mu,X)$. The multivariate Stein identity, under the stated conditions, gives

E_\mu[(X-\mu)^\mathsf{T}g(X)] = \sigma^2 E_\mu[\operatorname{div}g(X)].

Substituting this identity into the expanded risk and subtracting $R(\mu,X)$ proves the formula. $\square$

The two terms have distinct roles. The quantity $\|g(X)\|^2$ is the squared size of the adjustment at the observed data; it is always nonnegative. The divergence term describes how the adjustment changes locally as the data coordinates change. Its expectation can be positive, zero, or negative. A negative expected divergence can offset the squared cost, but it need not be large enough to do so.

Worked Applications

Worked Example: A Constant Shift in Two Dimensions

Let $X\sim N_2(\mu,4I_2)$, and use the constant adjustment $g(x)=(1,-2)^\mathsf{T}$. Compute the risk difference and the two contributions in the divergence formula.

The squared adjustment is constant:

\|g(x)\|^2=1^2+(-2)^2=5.

Each component of $g$ is constant, so both diagonal partial derivatives are zero and $\operatorname{div}g(x)=0$. Therefore,

R(\mu,\delta)-R(\mu,X)=E_\mu[5+2(4)(0)]=5.

The baseline risk is $p\sigma^2=2(4)=8$, so $R(\mu,\delta)=8+5=13$. This shift increases risk by 5 at every $\mu$. It has no divergence contribution to offset its squared cost.

Worked Example: A Sinusoidal Adjustment That Reduces Risk

Consider the scalar model $X\sim N(\mu,1)$ at $\mu=0$, with $g(x)=-0.5\sin x$. This smooth adjustment and its derivative are bounded, so the required integrability and boundary conditions hold. Calculate the risk change.

Here $g'(x)=-0.5\cos x$, and the divergence in one dimension is just $g'(x)$. For a standard normal variable, $E[\cos X]=e^{-1/2}$ and $E[\sin^2X]=(1-e^{-2})/2$. Thus the squared-adjustment contribution is

E[g(X)^2] = 0.25E[\sin^2 X] = 0.125(1-e^{-2}) \approx 0.108083.

The divergence contribution is

2E[g'(X)] = -E[\cos X] = -e^{-1/2} \approx -0.606531.

Adding the terms gives $R(0,\delta)-R(0,X)\approx 0.108083-0.606531=-0.498448$. Since the baseline risk is 1, the adjusted estimator has risk approximately $0.501552$ at $\mu=0$. In this calculation the negative expected divergence more than offsets the positive squared-adjustment cost. The conclusion is for this parameter value; this example alone does not establish a uniform risk comparison over all $\mu$.

Worked Example: A Smooth Localized Adjustment With Increased Risk

Let $X\sim N(0,1)$ and $g(x)=0.6e^{-x^2/2}$. This function and its derivative are integrable under the normal distribution and satisfy the boundary condition. Its derivative is $g'(x)=-0.6xe^{-x^2/2}$, which is odd, so $E[g'(X)]=0$ by symmetry.

The squared-adjustment term is

E[g(X)^2] = 0.36E[e^{-X^2}] = \frac{0.36}{\sqrt{3}} \approx 0.207846.

For the expectation, the standard normal density contributes a factor $e^{-x^2/2}$; multiplying by $e^{-x^2}$ gives an integral proportional to $e^{-3x^2/2}$, whose normalized expectation is $1/\sqrt{3}$. Consequently,

R(0,\delta)-R(0,X) = \frac{0.36}{\sqrt{3}}+2E[g'(X)] = \frac{0.36}{\sqrt{3}} \approx 0.207846.

The baseline risk is 1, so the adjusted risk is approximately $1.207846$. Although the derivative changes sign across the real line, its expectation is zero here; there is no negative average divergence to counteract the positive adjustment cost.

How to Read the Comparison

The formula is an expectation of a sum, not a claim that every observed value of $X$ produces a lower squared error. Its sign answers a risk question at a fixed parameter $\mu$: a negative value means the expected squared-error risk of $\delta$ is smaller than that of $X$ at that $\mu$; a positive value means it is larger.

The divergence is a local measure of how the adjustment field expands or contracts as the data vary. For example, in one dimension a decreasing adjustment has negative derivative where it decreases. But a negative derivative at some data values does not by itself determine the risk difference: the formula uses the expectation of the divergence together with the expected squared size of the entire adjustment. Nor does a negative divergence at a particular $x$ guarantee that the expectation of the sum is negative.

A common error is to stop after observing that $\operatorname{div}g(x)<0$ in part of the sample space and conclude that risk is reduced. The missing step is to account for both the squared cost and the distribution of $X$. The relevant comparison is exactly

E_\mu[\|g(X)\|^2]+2\sigma^2E_\mu[\operatorname{div}g(X)],

not the divergence alone and not the value of the expression at one chosen data point. Another frequent error is to omit the factor $2\sigma^2$: the factor 2 comes from the cross term in the squared-error expansion, and $\sigma^2$ comes from Gaussian integration by parts.

Key takeaway: The risk change from adding $g(X)$ to $X$ is the expected squared size of that adjustment plus twice the noise variance times its expected divergence. Risk falls only when the divergence contribution is negative enough to outweigh the squared cost.

Check Your Understanding

Use the formula and its assumptions to answer the following questions.

For a constant vector adjustment, what is its divergence, and what does the risk difference reduce to?
Why is the squared-adjustment contribution always nonnegative, while the divergence contribution can have either sign?
In the sinusoidal example, which contribution is larger in magnitude, and how does that determine the risk change at $\mu=0$?
Why is a negative divergence at one observed data point insufficient to establish a negative risk difference?
Identify the source of the factor $2\sigma^2$ in the divergence formula.

Divergence Formula for Risk Difference

What You'll Learn