Vector Quadratic Loss And Trace Risk

Why Vector Risk Has a Trace Form

When an estimator reports several coordinates, its error is itself a vector. Under ordinary squared Euclidean loss, the total loss is the sum of the squared coordinate errors. This makes the expected loss a single number, even when the estimator’s coordinates are dependent. The covariance matrix gives a compact way to collect the coordinate variances: their sum is its trace.

The resulting identity separates risk into two sources: variation of the estimator around its own mean and displacement of that mean from the target. It applies to any finite-dimensional estimator with finite second moments; it does not require a Gaussian model or independent coordinates.

Definition: Let $\theta\in\mathbb{R}^p$ be a fixed parameter vector and let $\delta=(\delta_1,\ldots,\delta_p)^T$ be a measurable estimator taking values in $\mathbb{R}^p$. Its ordinary quadratic loss at $\theta$ is $L(\theta,\delta)=\|\delta-\theta\|^2=(\delta-\theta)^T(\delta-\theta)$. Its risk is $R(\theta,\delta)=E_\theta[\|\delta-\theta\|^2]$, with the expectation allowed to be infinite.

Here $E_\theta$ means expectation under the distribution of the data when the parameter is $\theta$. For a vector $v$, $\|v\|^2=\sum_{j=1}^p v_j^2$. When the coordinates of $\delta$ have finite second moments, define its mean vector, bias vector, and covariance matrix by $m_\theta=E_\theta[\delta]$, $b_\theta=m_\theta-\theta$, and $\operatorname{Cov}_\theta(\delta)=E_\theta[(\delta-m_\theta)(\delta-m_\theta)^T]$.

The Trace-Risk Identity

Theorem: For fixed $\theta\in\mathbb{R}^p$, if $E_\theta[\|\delta\|^2]<\infty$, then $$ R(\theta,\delta) = \operatorname{tr}\!\left(\operatorname{Cov}_\theta(\delta)\right) + \|b_\theta\|^2. $$ Equivalently, the trace term is $\sum_{j=1}^p\operatorname{Var}_\theta(\delta_j)$, and the squared-bias term is $\sum_{j=1}^p(E_\theta[\delta_j]-\theta_j)^2$.

To prove the identity, write the estimation error as a centered random vector plus the bias: $\delta-\theta=(\delta-m_\theta)+b_\theta$. Squaring its Euclidean norm gives

\|\delta-\theta\|^2 = \|\delta-m_\theta\|^2 +2b_\theta^T(\delta-m_\theta) +\|b_\theta\|^2.

All terms are integrable under the stated second-moment condition. In particular, the cross term is integrable by the Cauchy–Schwarz inequality. Its expectation is zero because $E_\theta[\delta-m_\theta]=0$. Also, $\|\delta-m_\theta\|^2=\sum_{j=1}^p(\delta_j-m_{\theta,j})^2$, so its expectation is the sum of the coordinate variances. The trace of a matrix is the sum of its diagonal entries, and the diagonal entries of the covariance matrix are those variances. Taking expectations therefore gives the claimed identity. $\square$

This argument does not assume that different coordinates are independent. Their covariances appear as off-diagonal entries of the covariance matrix, but the trace does not include those entries. Under ordinary squared Euclidean loss, the loss is a sum of coordinatewise squares, with no cross-products between different coordinates.

Worked Example: A Correlated Gaussian Estimator

Suppose $p=2$, $\theta=(2,-1)^T$, and the estimator $\delta=(X_1,X_2)^T$ has mean and covariance $E_\theta[\delta]=(2.5,-2)^T$ and $\operatorname{Cov}_\theta(\delta)=\begin{pmatrix}4&1\\1&1\end{pmatrix}$. For example, these can be the mean and covariance of a bivariate normal estimator. The covariance matrix is valid: its diagonal entries are positive and its determinant is $4(1)-1=3>0$.

The bias is $b_\theta=(2.5-2,-2-(-1))^T=(0.5,-1)^T$, so its squared norm is $0.5^2+(-1)^2=1.25$. The covariance trace is $4+1=5$. Thus $R(\theta,\delta)=5+1.25=6.25$.

Checking by coordinates gives the same result. The first coordinate contributes $\operatorname{Var}(X_1)+(E[X_1]-2)^2=4+0.25=4.25$, and the second contributes $\operatorname{Var}(X_2)+(E[X_2]+1)^2=1+1=2$. Their sum is $4.25+2=6.25$. The covariance $1$ between the coordinates does not enter this ordinary quadratic risk.

When the Risk and Decomposition Are Finite

The second-moment condition is not a technical detail that can be omitted. It is exactly the condition needed for finite quadratic risk at a fixed, finite parameter vector. This equivalence also clarifies what to conclude when an estimator has heavy tails.

Proposition: For a fixed $\theta\in\mathbb{R}^p$, $R(\theta,\delta)<\infty$ if and only if $E_\theta[\|\delta\|^2]<\infty$. Under either condition, each coordinate has a finite second moment, so the mean vector and covariance matrix in the trace-risk identity are finite.

For one direction, the inequality $\|a+c\|^2\leq 2\|a\|^2+2\|c\|^2$, applied to $\delta=(\delta-\theta)+\theta$, gives

E_\theta[\|\delta\|^2] \leq 2E_\theta[\|\delta-\theta\|^2]+2\|\theta\|^2.

Thus finite risk implies a finite second moment for $\delta$. Conversely, apply the same inequality to $\delta-\theta$ to obtain

E_\theta[\|\delta-\theta\|^2] \leq 2E_\theta[\|\delta\|^2]+2\|\theta\|^2.

So a finite second moment implies finite risk. Finally, $\delta_j^2\leq\|\delta\|^2$ for every coordinate $j$, hence each coordinate has finite second moment. Its mean and variance are finite, as are all covariance entries by Cauchy–Schwarz. This establishes the proposition and the conditions under which the trace-risk identity is a finite-valued decomposition. $\square$

Worked Example: A Discrete Estimator With Correlated Coordinates

Let $\theta=(1,-2)^T$, and define $Z$ to equal $(1,1)^T$ or $(-1,-1)^T$, each with probability $1/2$. Set $\delta=\theta+(1,0)^T+Z$. The estimator takes the values $(3,-1)^T$ and $(1,-3)^T$, each with probability $1/2$.

Its mean is $\theta+(1,0)^T=(2,-2)^T$, so the bias is $(1,0)^T$ and the squared bias is $1$. Since $Z$ is centered and its coordinates are equal, its covariance matrix is $\begin{pmatrix}1&1\\1&1\end{pmatrix}$, whose trace is $2$. The identity gives risk $2+1=3$.

Directly, the two estimation errors are $(2,1)^T$ and $(0,-1)^T$. Their squared norms are $2^2+1^2=5$ and $0^2+(-1)^2=1$. The average loss is $(5+1)/2=3$, confirming the calculation. The coordinates are perfectly correlated here, but the off-diagonal covariance still is not added to the trace.

Worked Example: Finite Mean Does Not Guarantee Finite Risk

Let $Z$ have density $f(z)=\tfrac34|z|^{-5/2}$ for $|z|\geq1$, and density zero otherwise. This is a probability density because $2(\tfrac34)\int_1^\infty z^{-5/2}\,dz=(\tfrac32)(\tfrac23)=1$. It is symmetric, so its mean is zero once absolute integrability is checked.

The first absolute moment is finite: $E[|Z|]=2(\tfrac34)\int_1^\infty z^{-3/2}\,dz=(\tfrac32)(2)=3$. But the second moment is infinite: $E[Z^2]=2(\tfrac34)\int_1^\infty z^{-1/2}\,dz=\infty$. For $\theta=(0,0)^T$ and $\delta=(Z,0)^T$, the risk is $E_\theta[\|\delta-\theta\|^2]=E[Z^2]=\infty$.

The mean vector exists and the bias is zero, but a finite trace of the covariance matrix does not exist. It would be incorrect to report zero risk from the squared-bias term alone: the trace-risk identity as a finite decomposition requires finite second moments.

Common Pitfall and Practical Use

A common mistake is to include every covariance between coordinates when calculating risk, on the grounds that the coordinates are dependent. For ordinary Euclidean loss, expanding the loss directly shows why this is wrong: $\|\delta-\theta\|^2=\sum_{j=1}^p(\delta_j-\theta_j)^2$. There are no terms $(\delta_j-\theta_j)(\delta_k-\theta_k)$ for $j\ne k$, so their expectations cannot contribute to this loss. Dependence can matter for the distribution of the total error, but not through off-diagonal entries in its expected squared Euclidean norm.

The trace-plus-squared-bias identity is useful because it turns a vector risk calculation into two manageable summaries: total marginal variance and squared bias length. Before using it, check that the estimator has a finite second moment under the parameter value of interest. If that condition fails, the risk may be infinite, and the usual finite covariance decomposition is unavailable.

Check Your Understanding

Use the definitions and conditions in this tutorial to answer the following questions.

For a three-coordinate estimator with finite second moments, what entries of its covariance matrix contribute to the trace term?
If an estimator is unbiased and has finite second moments, how does its risk relate to the trace of its covariance matrix?
Why does a finite risk at fixed $\theta$ imply that every coordinate of the estimator has a finite second moment?
For the discrete estimator above, identify its bias vector and explain why the off-diagonal covariance does not alter the risk.
Can an estimator have a finite mean vector but infinite quadratic risk? Explain using the heavy-tailed example.

Vector Quadratic Loss And Trace Risk

What You'll Learn