Weighted Quadratic Loss Across Coordinates

Why Weight Coordinates Differently?

In a vector estimation problem, errors in different coordinates need not have the same practical importance. A one-unit error in one component might be more consequential than a one-unit error in another. A weighted quadratic loss makes this distinction explicit. It also allows the loss to account for interactions between coordinates, rather than treating each squared error in isolation.

The key calculation is a matrix version of the bias–variance identity. Risk separates into a term determined by the estimator’s mean error and a term determined by its covariance. The weight matrix acts on both terms, so changing the weights can change not only the total risk but also how much bias and variability contribute to it.

Weighted Quadratic Loss and Risk

Let the parameter of interest be a fixed vector $\theta\in\mathbb{R}^p$, and let $\hat{\theta}\in\mathbb{R}^p$ be an estimator with finite second moments under the distribution indexed by $\theta$. Let $W$ be a fixed, symmetric, positive-definite $p\times p$ matrix. The loss for estimating $\theta$ by $\hat{\theta}$ is the weighted quadratic loss below.

Definition: The weighted quadratic loss is $L_W(\theta,\hat{\theta})=(\hat{\theta}-\theta)^{T}W(\hat{\theta}-\theta)$. The corresponding risk is $R_W(\theta,\hat{\theta})=E_{\theta}[L_W(\theta,\hat{\theta})]$, when this expectation is finite. Positive definiteness means $v^{T}Wv>0$ for every nonzero vector $v$, so the loss is nonnegative and vanishes only when $\hat{\theta}=\theta$.

If $W$ is diagonal, its diagonal entries weight the squared errors in the individual coordinates. A nondiagonal $W$ also includes products of errors from different coordinates. Symmetry ensures that the quadratic form is real and that the cross-terms are combined consistently. Positive definiteness is a property of the whole matrix: the off-diagonal entries cannot be chosen arbitrarily while retaining a valid loss.

Worked Example: Diagonal Weights on Two Coordinates

Suppose the estimation error is $e=\hat{\theta}-\theta$, with $E_{\theta}[e]=(1,-1)^{T}$ and $\operatorname{Cov}_{\theta}(\hat{\theta})=\begin{pmatrix}1&0\\0&4\end{pmatrix}$. Use $W=\begin{pmatrix}1&0\\0&3\end{pmatrix}$ to calculate the risk.

Both diagonal entries of $W$ are positive, so it is positive definite. The loss is $e_1^2+3e_2^2$. The first coordinate has mean error 1 and variance 1, giving $E_{\theta}[e_1^2]=1^2+1=2$. The second has mean error $-1$ and variance 4, giving $E_{\theta}[e_2^2]=(-1)^2+4=5$. Therefore,

R_W(\theta,\hat{\theta})=E_{\theta}[e_1^2+3e_2^2] =2+3(5)=17.

The second coordinate contributes 15 units of risk, compared with 2 units from the first. Its larger contribution reflects both its larger variance and the greater loss weight assigned to that coordinate.

Bias–Covariance Decomposition

Write $b_{\theta}=E_{\theta}[\hat{\theta}]-\theta$ for the bias vector and $\Sigma_{\theta}=\operatorname{Cov}_{\theta}(\hat{\theta})$ for the covariance matrix. The following identity gives the central risk calculation. Finite second moments ensure that the expectations involved exist.

Theorem: For fixed symmetric $W$ and an estimator with finite second moments, $$ R_W(\theta,\hat{\theta})=b_{\theta}^{T}Wb_{\theta}+\operatorname{tr}(W\Sigma_{\theta}). $$

To prove the identity, let $e=\hat{\theta}-\theta$ and $z=e-b_{\theta}$. Then $E_{\theta}[z]=0$ and $E_{\theta}[zz^{T}]=\Sigma_{\theta}$. Since $e=b_{\theta}+z$, expanding the quadratic form gives

e^{T}We=b_{\theta}^{T}Wb_{\theta} +b_{\theta}^{T}Wz+z^{T}Wb_{\theta}+z^{T}Wz.

The expectation of each middle term is zero: $E_{\theta}[b_{\theta}^{T}Wz]=b_{\theta}^{T}WE_{\theta}[z]=0$, and $E_{\theta}[z^{T}Wb_{\theta}]=E_{\theta}[z]^{T}Wb_{\theta}=0$. For the final term, the scalar trace identity $z^{T}Wz=\operatorname{tr}(Wzz^{T})$ yields

E_{\theta}[z^{T}Wz] =\operatorname{tr}\bigl(W E_{\theta}[zz^{T}]\bigr) =\operatorname{tr}(W\Sigma_{\theta}).

Linearity of expectation and of the trace justifies taking the expectation inside this finite-dimensional expression. Combining the terms proves the stated formula. $\square$

The two contributions are both nonnegative. The bias term is nonnegative because $W$ is positive definite. For the covariance term, write $W=Q\Lambda Q^{T}$, where $Q$ is orthogonal and $\Lambda$ is diagonal with positive entries. Then $\operatorname{tr}(W\Sigma_{\theta})=\operatorname{tr}(\Lambda Q^{T}\Sigma_{\theta}Q)$; the diagonal entries of $Q^{T}\Sigma_{\theta}Q$ are variances and are nonnegative. Thus the trace contribution is nonnegative as well.

Worked Example: A Nondiagonal Weight and Correlated Errors

Suppose $b_{\theta}=(1,-1)^{T}$, $\Sigma_{\theta}=\begin{pmatrix}1&0.5\\0.5&2\end{pmatrix}$, and $W=\begin{pmatrix}2&1\\1&2\end{pmatrix}$. Calculate each term in the risk decomposition. The weight matrix has eigenvalues 1 and 3, so it is positive definite. The covariance matrix is also valid: its diagonal entries are positive and its determinant is $1(2)-(0.5)^2=1.75>0$.

For the bias contribution, first multiply $Wb_{\theta}=(1,-1)^{T}$. Hence $b_{\theta}^{T}Wb_{\theta}=(1,-1)(1,-1)^{T}=2$.

For the covariance contribution, the diagonal entries of $W\Sigma_{\theta}$ are $2(1)+1(0.5)=2.5$ and $1(0.5)+2(2)=4.5$. Thus $\operatorname{tr}(W\Sigma_{\theta})=2.5+4.5=7$, and

R_W(\theta,\hat{\theta})=2+7=9.

Here the off-diagonal weights make the covariance contribution depend on the covariance between the coordinate errors. In particular, the two symmetric cross-terms contribute $2(1)(0.5)=1$ to the trace. Ignoring that dependence would give the wrong risk.

What Diagonal Weights Do

When $W=\operatorname{diag}(w_1,\ldots,w_p)$ with each $w_j>0$, the loss is a sum of separately weighted coordinatewise squared errors. The risk therefore has a particularly direct expression.

Result: For diagonal $W=\operatorname{diag}(w_1,\ldots,w_p)$, $$ R_W(\theta,\hat{\theta}) =\sum_{j=1}^{p}w_j\left\{b_{\theta,j}^{\,2}+\operatorname{Var}_{\theta}(\hat{\theta}_j)\right\}. $$

For a derivation, the diagonal quadratic form is $e^{T}We=\sum_{j=1}^{p}w_je_j^2$. Taking expectations gives $R_W=\sum_{j=1}^{p}w_jE_{\theta}[e_j^2]$. For each coordinate, write $e_j=(e_j-E_{\theta}[e_j])+E_{\theta}[e_j]$. The centered term has mean zero, so expanding its square gives $E_{\theta}[e_j^2]=\operatorname{Var}_{\theta}(\hat{\theta}_j)+b_{\theta,j}^{\,2}$. Substitution proves the result. $\square$

This expression shows precisely how the weights change the contributions: coordinate $j$'s squared bias and variance are both multiplied by $w_j$. Correlations between coordinates do not enter when $W$ is diagonal, because the trace $\operatorname{tr}(W\Sigma_{\theta})$ uses only the diagonal entries of $\Sigma_{\theta}$.

Worked Example: Weighted Risk for Shrunk Normal Coordinates

Let $X_1\sim N(\theta_1,1)$ and $X_2\sim N(\theta_2,1)$ be independent. Estimate $\theta=(2,-1)^{T}$ using $\hat{\theta}=(0.5X_1,0.8X_2)^{T}$, and set $W=\operatorname{diag}(1,4)$. Calculate the risk under this fixed parameter value.

The estimator means are $E_{\theta}[\hat{\theta}]=(1,-0.8)^{T}$, so the bias vector is $(-1,0.2)^{T}$. Independence and the unit variances of $X_1,X_2$ give $\operatorname{Var}_{\theta}(\hat{\theta}_1)=0.5^2=0.25$, $\operatorname{Var}_{\theta}(\hat{\theta}_2)=0.8^2=0.64$, and zero covariance.

The weighted bias contribution is $1(-1)^2+4(0.2)^2=1+0.16=1.16$. The weighted covariance contribution is $1(0.25)+4(0.64)=0.25+2.56=2.81$. Therefore,

R_W(\theta,\hat{\theta})=1.16+2.81=3.97.

For a direct check, the coordinatewise mean squared errors are $1+0.25=1.25$ and $0.2^2+0.64=0.68$. Weighting them gives $1(1.25)+4(0.68)=1.25+2.72=3.97$, agreeing with the decomposition.

Interpretation and a Common Pitfall

A positive-definite weight matrix defines which directions of estimation error are costly. With diagonal weights, the interpretation is coordinatewise: a larger $w_j$ makes both bias and variability in coordinate $j$ count more heavily. With nondiagonal weights, the loss and risk can also involve interactions. For example, $e^{T}We$ includes $2W_{ij}e_ie_j$ when $i\ne j$, and the covariance term includes the corresponding interaction between errors.

A common mistake is to calculate unweighted coordinatewise mean squared errors, add them, and assume that the result is the risk for any quadratic loss. That sum is the risk only for $W=I$. With diagonal weights, each coordinate's contribution must be multiplied by its own weight; with nondiagonal weights, cross-coordinate terms must also be included. For instance, in the preceding nondiagonal example, keeping only weighted marginal variances would omit the contribution from covariance.

Key takeaway: Under weighted quadratic loss, risk is weighted squared bias plus weighted covariance, not simply a sum of unweighted coordinate risks. The matrix $W$ determines how both coordinate errors and their dependence affect the total.

Check Your Understanding

Use the loss definition and risk decomposition to answer the following.

For $W=\operatorname{diag}(2,5)$, what is the loss in terms of the two coordinate errors?
Why does the bias–covariance decomposition require finite second moments?
If $W$ is diagonal, which entries of the covariance matrix contribute to $\operatorname{tr}(W\Sigma_{\theta})$?
In two dimensions, where can the covariance between coordinate errors appear when $W$ is nondiagonal?
Suppose a diagonal weight on one coordinate is multiplied by 3. Which parts of that coordinate’s risk contribution are multiplied by 3?