Covariance Propagation For Linear Estimators

Why Covariance Propagation Matters

A linear estimator can mix several observed quantities, rescale them, or map them to a different-dimensional estimate. Its uncertainty is therefore not obtained by looking at each coordinate separately: the dependence among observations can change the uncertainty after they are combined. Matrix operations provide a direct way to keep track of both the transformed covariance and the estimator’s bias.

This tutorial derives those operations for an affine estimator. Once its mean and covariance are known, its quadratic risk follows from a bias-and-covariance decomposition. The formulas do not require normality; finite second moments are enough.

Setup and Definitions

For each parameter value $\theta$, let $X$ be a random observation in $\mathbb{R}^p$ with mean vector $m(\theta)=E_\theta[X]$ and covariance matrix $\Sigma(\theta)=\operatorname{Cov}_\theta(X)$. Assume these quantities exist and are finite. The target is a vector $\theta\in\mathbb{R}^k$, and the estimator has the affine form $\hat{\theta}=AX+b$, where $A$ is a fixed $k\times p$ matrix and $b$ is a fixed $k$-vector. The choices of $A$ and $b$ may depend on known features of the problem, but not on the observed value of $X$.

Definition: The bias vector of $\hat{\theta}$ at $\theta$ is $E_\theta[\hat{\theta}]-\theta$. Its covariance matrix is $\operatorname{Cov}_\theta(\hat{\theta})$. Under a quadratic loss with a fixed symmetric positive semidefinite $k\times k$ matrix $W$, the loss is $L(\theta,\hat{\theta})=(\hat{\theta}-\theta)^T W(\hat{\theta}-\theta)$, and the risk is $R(\theta,\hat{\theta})=E_\theta[L(\theta,\hat{\theta})]$.

Positive semidefiniteness of $W$ ensures that the loss is nonnegative. The identity matrix gives ordinary squared Euclidean error. A diagonal $W$ assigns different weights to coordinates, while off-diagonal entries can make the loss sensitive to combinations of coordinates. The risk formula below applies when the estimator has finite second moments, which follows here from the assumed finite second moments of $X$.

Mean, Covariance, and Bias of an Affine Estimator

Theorem: If $E_\theta[X]=m(\theta)$ and $\operatorname{Cov}_\theta(X)=\Sigma(\theta)$ are finite, then for $\hat{\theta}=AX+b$, $$ E_\theta[\hat{\theta}]=A m(\theta)+b,\qquad \operatorname{Cov}_\theta(\hat{\theta})=A\Sigma(\theta)A^T, $$ and $$ \operatorname{Bias}_\theta(\hat{\theta})=A m(\theta)+b-\theta. $$

To verify the mean, use linearity of expectation: $E_\theta[AX+b]=A E_\theta[X]+b$. For the covariance, subtract the estimator’s mean:

\hat{\theta}-E_\theta[\hat{\theta}] =A\{X-m(\theta)\}.

Taking the expected outer product gives

\begin{aligned} \operatorname{Cov}_\theta(\hat{\theta}) &=E_\theta\!\left[ \{\hat{\theta}-E_\theta[\hat{\theta}]\} \{\hat{\theta}-E_\theta[\hat{\theta}]\}^T \right]\\ &=E_\theta\!\left[ A\{X-m(\theta)\}\{X-m(\theta)\}^T A^T \right]\\ &=A\Sigma(\theta)A^T. \end{aligned}

The last equality moves the fixed matrices outside the expectation. Subtracting $\theta$ from the mean gives the stated bias. In particular, a nonzero $b$ affects the mean and bias, but not the covariance. $\square$

Worked Example: A Two-Coordinate Transformation

Suppose $X\in\mathbb{R}^2$ has mean $m=(2,-1)^T$ and covariance $\Sigma=\begin{pmatrix}4&1\\1&2\end{pmatrix}$. Estimate $\theta=(1,0)^T$ with $\hat{\theta}=AX+b$, where $A=\begin{pmatrix}1&1\\0&2\end{pmatrix}$ and $b=(0,1)^T$. Find its mean, bias, covariance, and risk under $W=\begin{pmatrix}1&0\\0&2\end{pmatrix}$.

First, the mean is

E[\hat{\theta}] =Am+b =\begin{pmatrix}1&1\\0&2\end{pmatrix} \begin{pmatrix}2\\-1\end{pmatrix} +\begin{pmatrix}0\\1\end{pmatrix} =\begin{pmatrix}1\\-1\end{pmatrix}.

Thus the bias is $(1,-1)^T-(1,0)^T=(0,-1)^T$. For the covariance, the rows of $A$ show that the two estimated coordinates fluctuate as $X_1+X_2$ and $2X_2$. Their variances and covariance are $\operatorname{Var}(X_1+X_2)=4+2(1)+2=8$, $\operatorname{Cov}(X_1+X_2,2X_2)=2(1+2)=6$, and $\operatorname{Var}(2X_2)=4(2)=8$. Therefore

A\Sigma A^T=\begin{pmatrix}8&6\\6&8\end{pmatrix}.

For this diagonal weight matrix, the covariance contribution to risk is $8+2(8)=24$, and the squared weighted bias is $0^2+2(-1)^2=2$. The total risk is $26$.

Quadratic Risk as a Trace Plus a Bias Term

Theorem: Under the preceding finite-second-moment conditions, for symmetric $W$, $$ R(\theta,\hat{\theta}) =\operatorname{tr}\!\left(W A\Sigma(\theta)A^T\right) +\{A m(\theta)+b-\theta\}^T W\{A m(\theta)+b-\theta\}. $$ If $W$ is positive semidefinite, this is the risk under a nonnegative quadratic loss.

Let $e=\hat{\theta}-\theta$, let $\beta=E_\theta[e]$, and let $V=\operatorname{Cov}_\theta(\hat{\theta})$. By the definition of covariance, $E_\theta[ee^T]=V+\beta\beta^T$. Also, for any vector $e$, the scalar $e^TWe$ equals $\operatorname{tr}(Wee^T)$. Hence

\begin{aligned} R(\theta,\hat{\theta}) &=E_\theta[e^TWe]\\ &=\operatorname{tr}\!\left(W E_\theta[ee^T]\right)\\ &=\operatorname{tr}(WV)+\operatorname{tr}(W\beta\beta^T)\\ &=\operatorname{tr}(WV)+\beta^T W\beta. \end{aligned}

The trace and expectation can be interchanged here because the matrices are finite-dimensional and the entries of $ee^T$ have finite expectations. The final equality follows by expanding the trace of the rank-one matrix product. Substituting $V=A\Sigma(\theta)A^T$ and $\beta=A m(\theta)+b-\theta$ proves the formula. $\square$

This expression separates two sources of risk. The trace term measures variability after transformation; the other term measures systematic displacement of the estimator’s mean from the target. Both terms are nonnegative when $W$ is positive semidefinite, although the bias vector itself can have positive and negative coordinates.

Worked Example: A Weighted Average of Correlated Measurements

Let $X=(X_1,X_2,X_3)^T$ have mean $(\theta,\theta,\theta)^T$ and covariance $\Sigma=\begin{pmatrix}4&1&0\\1&9&2\\0&2&4\end{pmatrix}$. Consider the estimator $\hat{\theta}=(X_1+X_2)/2$ of the scalar $\theta$, with ordinary squared-error loss.

Here $A=(1/2,1/2,0)$, $b=0$, and the target dimension is one. Its mean is $A E[X]=\theta/2+\theta/2=\theta$, so its bias is zero. Its variance, and therefore its risk, is

\begin{aligned} A\Sigma A^T &=\frac14\operatorname{Var}(X_1+X_2)\\ &=\frac14\{\operatorname{Var}(X_1)+\operatorname{Var}(X_2) +2\operatorname{Cov}(X_1,X_2)\}\\ &=\frac14(4+9+2)=\frac{15}{4}=3.75. \end{aligned}

If the covariance between $X_1$ and $X_2$ were incorrectly omitted, the calculation would give $(4+9)/4=3.25$, which is too small. The observations’ positive covariance contributes $2/4=0.5$ to the variance of their average.

Using Coordinate Weights Correctly

For ordinary squared Euclidean loss, $W=I_k$, so risk is $\operatorname{tr}(A\Sigma A^T)+\|\operatorname{Bias}_\theta(\hat{\theta})\|^2$. With diagonal weights $W=\operatorname{diag}(w_1,\ldots,w_k)$, the trace term is a weighted sum of the estimator’s coordinate variances:

\operatorname{tr}(W\operatorname{Cov}_\theta(\hat{\theta})) =\sum_{j=1}^k w_j\operatorname{Var}_\theta(\hat{\theta}_j).

The covariance between different estimated coordinates does not appear in this particular trace when $W$ is diagonal, but it may still have affected each coordinate’s variance through $A\Sigma A^T$. If $W$ has off-diagonal entries, the trace also includes weighted cross-covariance terms. Thus “off-diagonal covariance does not matter” is not generally correct: its relevance depends on both the transformation $A$ and the loss matrix $W$.

Worked Example: Nonzero Bias and Weighted Risk

Suppose $X\in\mathbb{R}^2$ has mean $(\theta_1,\theta_2)^T$ and covariance $\Sigma=\begin{pmatrix}1&-1/2\\-1/2&4\end{pmatrix}$. Estimate $\theta$ by $\hat{\theta}=AX$, where $A=\begin{pmatrix}1&0\\1/2&1/2\end{pmatrix}$, and use $W=\operatorname{diag}(1,3)$. Calculate the bias and risk at $\theta=(2,0)^T$.

The mean of the estimator is $A\theta=(2,1)^T$, so the bias is $(0,1)^T$. Its weighted squared bias is $(0,1)W(0,1)^T=3$. The estimator’s first coordinate is $X_1$, with variance $1$. Its second coordinate is $(X_1+X_2)/2$, with variance

\operatorname{Var}\left(\frac{X_1+X_2}{2}\right) =\frac14\left(1+4+2\left(-\frac12\right)\right) =1.

Since $W$ is diagonal, the weighted variance contribution is $1+3(1)=4$. Adding the weighted squared bias gives $R(\theta,\hat{\theta})=4+3=7$. Notice that ignoring the negative covariance would give variance $5/4$ for the second coordinate and an incorrect risk of $4.75+3=7.75$.

Common Mistakes and Practical Checks

The formulas are compact, but dimension errors or missing covariance terms can change a result substantially. Before interpreting a calculation, check that $A$ has $k$ rows and $p$ columns, $\Sigma$ is $p\times p$, and $A\Sigma A^T$ is $k\times k$. The bias must be a $k$-vector, and $W$ must be $k\times k$, so both terms in the risk formula are scalars.

Dropping the affine offset from the mean. The mean is $A m(\theta)+b$, not just $A m(\theta)$. The offset has no effect on covariance, but it can change bias and risk.
Transforming covariance on only one side. The covariance is $A\Sigma A^T$, not $A\Sigma$. The transpose is required both for the dimensions and for the resulting matrix to be symmetric.
Assuming unbiasedness from a linear-looking formula. The estimator is unbiased only if $A m(\theta)+b=\theta$ at the parameter values in question.
Discarding correlations between observed coordinates. Off-diagonal entries of $\Sigma$ contribute whenever the estimator combines those coordinates.
Using a trace term without the loss weights. Under weighted loss, the variance contribution is $\operatorname{tr}(W\operatorname{Cov}_\theta(\hat{\theta}))$, not generally just the trace of the covariance.

These calculations matter whenever a procedure combines several measurements or estimates several targets at once. They allow a risk calculation to be assembled from the observation’s mean and covariance without requiring a full distributional derivation. In particular, normality is not needed for these affine-estimator identities; it becomes relevant only for questions that depend on distributional features beyond the first two moments.

Key takeaway: For an affine estimator $AX+b$, propagate the observation moments as $E[\hat{\theta}]=A m+b$ and $\operatorname{Cov}(\hat{\theta})=A\Sigma A^T$. Then quadratic risk is the weighted covariance trace plus the weighted squared bias.

Check Your Understanding

Use the formulas above, keeping track of dimensions and covariance terms.

If $X\in\mathbb{R}^p$ has covariance $\Sigma$ and $\hat{\theta}=AX+b$, what is the dimension and formula for $\operatorname{Cov}(\hat{\theta})$?
State the condition for this affine estimator to be unbiased at a specified $\theta$.
Why can positive covariance between two observations increase the variance of their average?
For a diagonal weight matrix $W$, which entries of $\operatorname{Cov}(\hat{\theta})$ appear directly in $\operatorname{tr}(W\operatorname{Cov}(\hat{\theta}))$?
Does the mean-covariance-risk derivation require $X$ to be normally distributed? State the moment condition it does require.