Minimum-Variance Combinations Of Correlated Estimators

Why Covariance Matters When Combining Estimators

Suppose two procedures estimate the same scalar parameter \(\theta\). If their errors tend to move together, averaging their answers may provide much less variance reduction than it would if their errors were independent. If their errors tend to move in opposite directions, a combination can sometimes be more precise than either estimator alone. The relevant quantity is not just each estimator’s variance, but also the covariance between their errors.

This tutorial derives the weights that minimize asymptotic variance among unbiased linear combinations of two estimators. The weights are unrestricted real numbers unless a restriction is stated explicitly; in particular, they are not automatically required to be nonnegative.

The matrix \(\begin{pmatrix}v_1&c\\c&v_2\end{pmatrix}\) must be positive semidefinite, since it is a limit of covariance matrices. In particular, \(v_1\geq 0\), \(v_2\geq 0\), and \(c^2\leq v_1v_2\). These restrictions are useful checks on any numerical example.

If, in addition, the centered and scaled pair converges jointly to a normal distribution with this covariance matrix, then the combined estimator has the corresponding normal limit for each fixed \(w\). The variance minimization below only needs the stated covariance limits; the joint normal limit gives the usual distributional interpretation.

Deriving the Optimal Weight

Write \(D=v_1+v_2-2c\). This quantity is the asymptotic variance of the difference between the estimators: $$ D=\lim_{n\to\infty}n\operatorname{Var}(T_{1n}-T_{2n}). $$ When \(D>0\), the variance as a function of \(w\) is a strictly convex quadratic.

Proof. Expand the variance quadratic and differentiate: $$ V(w)=w^2v_1+(1-2w+w^2)v_2+2w(1-w)c $$ $$ =w^2(v_1+v_2-2c)+2w(c-v_2)+v_2. $$ Therefore \(V'(w)=2wD+2(c-v_2)\), and setting this derivative to zero gives \(w^*=(v_2-c)/D\). Since \(V''(w)=2D>0\), this stationary point is the unique minimum. Completing the square gives $$ V(w)=D\left(w-\frac{v_2-c}{D}\right)^2+\frac{v_1v_2-c^2}{D}, $$ which establishes the stated minimum. The numerator is nonnegative because the covariance matrix is positive semidefinite. \(\square\)

The weights add to one, but need not both lie between zero and one. In the special case \(c=0\), the independent-estimator formula follows: \(w^*=v_2/(v_1+v_2)\), so the smaller-variance estimator receives the larger weight. With nonzero covariance, that rule alone is not enough: the covariance changes both the optimal weight and the variance achieved.

Worked Calculations

How the Constraint on Weights Changes the Answer

The condition that weights sum to one is what preserves unbiasedness when both component estimators have expectation \(\theta\). It does not, by itself, impose \(0\leq w\leq 1\). If a procedure requires a convex combination, meaning both weights must be nonnegative, the preceding unconstrained answer is used only when \(w^*\) lies in \([0,1]\). Otherwise the constrained minimum occurs at the nearer endpoint: \(w=0\) or \(w=1\), whichever has smaller variance. This follows because \(V(w)\) is a convex quadratic with its unrestricted minimum at \(w^*\).

For example, in the negative-weight example, \(w^*=4/3\) is outside \([0,1]\). Under a convex-weight restriction, the minimum is at an endpoint. The endpoint variances are \(V(0)=9\) and \(V(1)=4\), so the best permitted convex combination is just the first estimator, with variance \(4\). The unrestricted linear combination improves on it by allowing the negative coefficient.

Degeneracy, Interpretation, and Common Pitfalls

If \(D=0\), then the limiting variance of \(T_{1n}-T_{2n}\), on the \(\sqrt n\) scale, is zero. In that case the quadratic has no curvature, and the displayed formula for \(w^*\) is undefined. Positive semidefiniteness forces \(v_1=v_2=c\): indeed, \(D=0\) implies \(v_1+v_2=2c\), while \(c^2\leq v_1v_2\) gives \(c\leq\sqrt{v_1v_2}\leq (v_1+v_2)/2\), so equality holds throughout. Consequently \(V(w)=v_1\) for every fixed \(w\). There is no unique variance-minimizing weight at this asymptotic order.

The covariance term matters whenever both estimators are calculated from related data or share sources of error. Treating them as independent without justification can lead to the wrong weights and the wrong claimed precision. Conversely, a covariance matrix must be feasible: if proposed values violate \(c^2\leq v_1v_2\), they cannot be the variances and covariance of a pair of random variables.

• Assuming independence. Setting \(c=0\) is appropriate only when the covariance is zero. Shared observations often make the covariance nonzero.
• Forgetting the cross term. The variance includes \(2w(1-w)c\), not just the two weighted variances.
• Requiring positive weights without a stated restriction. Unrestricted linear combinations can have negative weights and can be more precise as a result.
• Using weights that do not sum to one. If both estimators are unbiased for \(\theta\), weights \(a,b\) produce expectation \((a+b)\theta\); unbiasedness for general \(\theta\) requires \(a+b=1\).