Diagnosing Errors In Quadratic-Risk Comparisons

Why Risk Comparisons Need an Audit

A numerical claim that one estimator has “smaller risk” is incomplete until the loss, parameter value, and averaging operation are clear. A calculation may omit bias, treat correlated observations as independent, compare errors in different units, or average over parameter values while sounding like a pointwise guarantee. These errors can change a conclusion, not just its presentation.

This tutorial develops a compact set of checks for quadratic-risk comparisons. The parameter is fixed when defining risk; randomness comes from the data. A comparison is pointwise if it is valid at each parameter value under consideration. If it averages risk across parameter values, that averaging rule must be stated separately.

Definition: Let $X$ have distribution indexed by $\theta$, let $\delta(X)$ estimate a vector parameter $\theta\in\mathbb{R}^p$, and let $W$ be a fixed symmetric positive semidefinite matrix. Under quadratic loss $L(\theta,\delta)=(\delta-\theta)^\mathsf{T}W(\delta-\theta)$, the risk is $R(\theta,\delta)=E_\theta[L(\theta,\delta)]$, provided the expectation exists. For scalar estimation with squared-error loss, take $p=1$ and $W=1$.

Separate Bias and Variance Before Comparing

For a scalar estimator with finite second moment, risk is not just variance unless the estimator is unbiased. Its bias at $\theta$ is $b(\theta)=E_\theta[\hat{\theta}]-\theta$, and its variance is $\operatorname{Var}_\theta(\hat{\theta})$.

Theorem: If $E_\theta[\hat{\theta}^2]<\infty$, then under squared-error loss, $$ R(\theta,\hat{\theta})=\operatorname{Var}_\theta(\hat{\theta})+b(\theta)^2. $$

To verify the identity, write $\hat{\theta}-\theta=(\hat{\theta}-E_\theta[\hat{\theta}])+b(\theta)$. Squaring and taking expectations gives

\begin{aligned} E_\theta[(\hat{\theta}-\theta)^2] &=E_\theta[(\hat{\theta}-E_\theta[\hat{\theta}])^2] +2b(\theta)E_\theta[\hat{\theta}-E_\theta[\hat{\theta}]] +b(\theta)^2\\ &=\operatorname{Var}_\theta(\hat{\theta})+b(\theta)^2, \end{aligned}

because the centered estimator has expectation zero. Thus a variance reduction alone does not establish a risk reduction: the squared bias must also be included.

Worked Example: A Biased Estimator Can Win Locally

Let $X\sim N(\theta,1)$. Compare the unbiased estimator $\delta_0(X)=X$ with the shrinkage estimator $\delta_1(X)=0.8X$, using squared-error loss.

For $\delta_0$, the bias is zero and the variance is 1, so $R(\theta,\delta_0)=1$. For $\delta_1$, its expectation is $0.8\theta$, giving bias $-0.2\theta$. Its variance is $0.8^2\cdot1=0.64$. Therefore

R(\theta,\delta_1)=0.64+(-0.2\theta)^2=0.64+0.04\theta^2.

At $\theta=0$, the risks are $0.64$ and $1$, so $\delta_1$ has lower risk. At $\theta=4$, its risk is $0.64+0.04(16)=1.28$, which exceeds the risk 1 of $\delta_0$. Saying “shrinkage lowers variance, therefore it lowers risk” misses the parameter-dependent squared bias.

Keep Covariance When Combining Data

The variance of a linear combination depends on covariances as well as marginal variances. For two random variables $U,V$ with finite second moments and constants $a,b$,

\operatorname{Var}(aU+bV) =a^2\operatorname{Var}(U)+b^2\operatorname{Var}(V)+2ab\operatorname{Cov}(U,V).

The covariance term is zero only under an appropriate condition, such as independence or zero covariance; it should not be discarded merely because two observations are written separately.

Worked Example: Averaging Correlated Measurements

Suppose $X_1=\theta+\varepsilon_1$ and $X_2=\theta+\varepsilon_2$, where $E[\varepsilon_i]=0$, $\operatorname{Var}(\varepsilon_i)=1$, and $\operatorname{Cov}(\varepsilon_1,\varepsilon_2)=0.6$. Compare $\delta=(X_1+X_2)/2$ with $X_1$ under squared-error loss.

The average is unbiased, so its risk is its variance. Using the covariance term,

R(\theta,\delta) =\operatorname{Var}\!\left(\frac{X_1+X_2}{2}\right) =\frac{1+1+2(0.6)}{4} =\frac{3.2}{4}=0.8.

The risk of $X_1$ is 1. The average therefore has risk 0.8, an improvement of 0.2. If the observations were incorrectly treated as independent, the calculation would give $(1+1)/4=0.5$, exaggerating the improvement. The conclusion that averaging helps happens to remain true here, but the reported risk is wrong.

Quadratic Risk for Vector Estimators

For vector estimation, the same bias-versus-variance check has a matrix form. It also helps reveal when covariance terms do and do not contribute. Write $b(\theta)=E_\theta[\delta]-\theta$ and $\Sigma(\theta)=\operatorname{Cov}_\theta(\delta)$.

Theorem: If the estimator has finite second moments and $W$ is fixed, symmetric, and positive semidefinite, then $$ R(\theta,\delta)=\operatorname{tr}\!\bigl(W\Sigma(\theta)\bigr)+b(\theta)^\mathsf{T}Wb(\theta). $$

To prove this, set $Z=\delta-E_\theta[\delta]$, so $E_\theta[Z]=0$ and $\delta-\theta=Z+b(\theta)$. Expanding the quadratic loss and taking expectations gives

\begin{aligned} R(\theta,\delta) &=E_\theta[(Z+b)^\mathsf{T}W(Z+b)]\\ &=E_\theta[Z^\mathsf{T}WZ]+b^\mathsf{T}Wb, \end{aligned}

since both cross terms have expectation zero. The identity $E[Z^\mathsf{T}WZ]=\operatorname{tr}(W E[ZZ^\mathsf{T}])$, obtained by expanding the finite sums of entries, gives $\operatorname{tr}(W\Sigma)$. This proves the result. $\square$

Under ordinary unweighted squared Euclidean loss, $W=I$, the variance contribution is $\operatorname{tr}(\Sigma)$, the sum of coordinate variances. Off-diagonal covariances do not appear directly in that trace. With non-diagonal $W$, or when a scalar estimator combines multiple correlated observations, covariance can affect risk. The loss and the estimator’s construction determine which covariance terms matter.

Check the Loss Scale and the Parameter Units

Multiplying one fixed loss function by a positive constant multiplies every risk under that loss by the same constant. It cannot reverse a comparison between two estimators when both are evaluated using the same loss. But changing the parameterization changes the quantity being estimated and may change the numerical scale of squared error.

Worked Example: Rescaling the Target Changes Squared Error

Let $X\sim N(\theta,1)$, and use $X$ to estimate $\theta$. Its squared-error risk is $E_\theta[(X-\theta)^2]=1$. Now define a new target $\eta=2\theta$. The corresponding estimator obtained by transforming $X$ is $2X$, and its risk for estimating $\eta$ is

E_\theta[(2X-\eta)^2] =E_\theta[(2X-2\theta)^2] =4E_\theta[(X-\theta)^2] =4.

The risks 1 and 4 are expressed in different squared units: squared units of $\theta$ and squared units of $\eta$. They are not evidence that the transformed estimator is intrinsically worse. A valid comparison requires the same target and loss, or an explicit account of how the loss changes under reparameterization. By contrast, multiplying both estimators’ loss by the same constant, such as replacing squared error by three times squared error, preserves their ranking.

Pointwise Risk Is Not Average Risk

A pointwise comparison checks $R(\theta,\delta_1)$ and $R(\theta,\delta_0)$ at each fixed $\theta$. An average-risk comparison first specifies a probability distribution or other weighting over parameter values and then averages the risk. An average advantage does not imply an advantage at every parameter value.

Result: If $R(\theta,\delta_1)\leq R(\theta,\delta_0)$ for every $\theta$ in a parameter space, then for any probability distribution $\pi$ on that space for which both integrated risks exist, $$ \int R(\theta,\delta_1)\,\pi(d\theta) \leq \int R(\theta,\delta_0)\,\pi(d\theta). $$

This follows by integrating the nonnegative function $R(\theta,\delta_0)-R(\theta,\delta_1)$ with respect to $\pi$. The reverse implication does not follow: a nonnegative average difference can coexist with a negative difference at some parameter values. The example below verifies this distinction explicitly.

Worked Example: Better on Average, Worse at One Parameter Value

Again let $X\sim N(\theta,1)$. Compare $\delta_0=X$, with risk 1 at every $\theta$, to $\delta_1=0.5X$. The latter has bias $-0.5\theta$ and variance $0.25$, so

R(\theta,\delta_1)=0.25+0.25\theta^2.

At $\theta=0$, this risk is 0.25, below 1. At $\theta=2$, it is $0.25+0.25(4)=1.25$, above 1. Now assign probability $1/2$ to each of the two parameter values 0 and 2. The average risks are

\frac{R(0,\delta_1)+R(2,\delta_1)}{2} =\frac{0.25+1.25}{2}=0.75, \qquad \frac{R(0,\delta_0)+R(2,\delta_0)}{2}=1.

Thus $\delta_1$ has lower average risk for this specified weighting, but it does not have lower risk at every parameter value. Reporting only the average would be misleading if the claim were meant to be pointwise.

A Direct Check for Comparing Two Rules

When two estimators are close in form, a direct expansion can expose the cross term that a shortcut misses. Suppose $\delta_1=\delta_0+h$, where $h$ is a vector-valued function of the data, and both rules have finite risk under the same quadratic loss.

Identity: For fixed symmetric $W$, $$ R(\theta,\delta_1)-R(\theta,\delta_0) =E_\theta\!\left[2(\delta_0-\theta)^\mathsf{T}Wh+h^\mathsf{T}Wh\right]. $$

Indeed, for each data outcome, substitute $\delta_1-\theta=(\delta_0-\theta)+h$ into the quadratic loss. Symmetry of $W$ makes the two cross terms equal, so their sum is $2(\delta_0-\theta)^\mathsf{T}Wh$. Taking expectations proves the identity. $\square$

The cross term can be positive or negative, so the extra quadratic term $h^\mathsf{T}Wh$ alone does not determine whether the new rule improves risk. This identity is a diagnostic tool, not a guarantee of improvement: the expectation must still be evaluated under the model and parameter value at issue.

Common Mistakes to Avoid

Using variance as risk for a biased rule. Add squared bias; the risk decomposition requires both terms.
Dropping covariance without justification. Check whether the relevant variables are uncorrelated, or calculate the covariance contribution explicitly.
Comparing numbers in different loss scales. Identify the target and loss first. A common positive multiplier preserves a ranking; a changed target or parameterization may not.
Turning an averaged claim into a universal claim. State the parameter weighting used and do not infer pointwise dominance from a lower average.
Leaving parameter dependence implicit. Bias and risk can depend on $\theta$; evaluate the comparison across the parameter values required by the claim.

Key takeaway: Before accepting a quadratic-risk comparison, identify the target and loss, calculate bias and covariance correctly, and state whether the conclusion holds pointwise or only after a specified averaging over parameter values.

Check Your Understanding

For each question, identify the calculation or claim that needs checking.

An estimator has variance 2 and bias 1 at a parameter value. What is its squared-error risk there?
Two measurements each have variance 1 and covariance 0.4. What is the variance of their average?
Why does multiplying the same loss by a positive constant preserve an estimator ranking, while changing the target from $\theta$ to $2\theta$ can change reported risk?
One rule has lower risk at $\theta=0$ and higher risk at $\theta=2$, but lower risk averaged equally over those two values. Which type of comparison has been established?
Under unweighted squared Euclidean loss, which part of the covariance matrix contributes directly to the variance term in vector risk?