Optimal Linear Weights From the Prediction-Error Covariance

Why the Error Covariance Determines the Weights

An ensemble combines predictions from several members, but equal weighting is not always optimal. A member with large errors may deserve less weight, while two members with strongly related errors may contribute less new information than their individual risks suggest. The relevant quantity is not covariance alone: systematic bias also affects the squared prediction error.

Let $f_j(X)$ be the prediction from member $j$, for $j=1,\ldots,m$, and let $Y$ be the response at the same prediction point. Define the error of member $j$ as $e_j=f_j(X)-Y$. We study deterministic weights $w=(w_1,\ldots,w_m)^\mathsf{T}$ whose entries sum to one. This restriction makes the ensemble error exactly the weighted sum of the member errors. The weights are otherwise unrestricted in sign: in this tutorial, “unconstrained” means that we do not impose nonnegativity or other inequality constraints.

Definition: Write $e=(e_1,\ldots,e_m)^\mathsf{T}$, let $b=E[e]$ be the vector of member biases, and let $C=\operatorname{Cov}(e)$ be the prediction-error covariance matrix. The error second-moment matrix is $$ M=E[ee^\mathsf{T}]=C+bb^\mathsf{T}. $$ For weights satisfying $\mathbf{1}^\mathsf{T}w=1$, the ensemble prediction is $\sum_{j=1}^m w_j f_j(X)$, and its squared-error risk is $R(w)=E[(\sum_{j=1}^m w_j f_j(X)-Y)^2]$.

Here $\mathbf{1}$ denotes the $m$-vector of ones. Since the weights sum to one, $\sum_j w_j f_j(X)-Y=\sum_j w_j(f_j(X)-Y)=w^\mathsf{T}e$. Thus the risk is the second moment of a weighted error, not merely the variance of that error. The matrix $M$ is positive semidefinite because $a^\mathsf{T}Ma=E[(a^\mathsf{T}e)^2]\geq 0$ for every vector $a$.

Writing Risk as a Quadratic Function

The covariance matrix records fluctuations around the member biases. Expanding the second moment separates these two contributions:

R(w)=E[(w^\mathsf{T}e)^2] =w^\mathsf{T}E[ee^\mathsf{T}]w =w^\mathsf{T}Cw+(w^\mathsf{T}b)^2 =w^\mathsf{T}Mw,\qquad \mathbf{1}^\mathsf{T}w=1.

The squared bias of the combined prediction is $(w^\mathsf{T}b)^2$, not the weighted sum of the members’ squared biases. Biases can partly cancel, so the optimal weights can differ substantially from those obtained by minimizing variance alone.

Worked Example: Two Members With Different Biases and Variances

Suppose two member errors have biases $b_1=0.2$ and $b_2=-0.1$, variances $1$ and $0.25$, and covariance $0$. The covariance matrix and error second-moment matrix are

C=\begin{pmatrix}1&0\\0&0.25\end{pmatrix}, \qquad bb^\mathsf{T}=\begin{pmatrix}0.04&-0.02\\-0.02&0.01\end{pmatrix}, \qquad M=\begin{pmatrix}1.04&-0.02\\-0.02&0.26\end{pmatrix}.

Write the weights as $(w_1,1-w_1)$. For a symmetric two-by-two matrix with entries $M_{11},M_{12},M_{22}$, differentiating the quadratic risk gives $w_1=(M_{22}-M_{12})/(M_{11}+M_{22}-2M_{12})$. Here,

w_1=\frac{0.26-(-0.02)}{1.04+0.26-2(-0.02)} =\frac{0.28}{1.34}=\frac{14}{67}, \qquad w_2=1-w_1=\frac{53}{67}.

The minimum risk is $w^\mathsf{T}Mw=\det(M)/(M_{11}+M_{22}-2M_{12})$. The determinant is $1.04(0.26)-(-0.02)^2=0.27$, so the minimum risk is $0.27/1.34=27/134$, approximately $0.2015$. The biases enter through the off-diagonal as well as the diagonal elements of $M$.

The Optimal Weights and Their Uniqueness

When $M$ is invertible, the constrained quadratic minimization has a closed-form solution. The formula follows from a Lagrange multiplier, with the sum-to-one condition retained throughout.

Theorem: Suppose $M$ is the error second-moment matrix and is positive definite. Among all $w\in\mathbb{R}^m$ satisfying $\mathbf{1}^\mathsf{T}w=1$, the unique minimizer of $R(w)=w^\mathsf{T}Mw$ is $$ w^*=\frac{M^{-1}\mathbf{1}}{\mathbf{1}^\mathsf{T}M^{-1}\mathbf{1}}, \qquad R(w^*)=\frac{1}{\mathbf{1}^\mathsf{T}M^{-1}\mathbf{1}}. $$

Proof. Define the Lagrangian $L(w,\lambda)=w^\mathsf{T}Mw-\lambda(\mathbf{1}^\mathsf{T}w-1)$. Its stationary equations are $2Mw-\lambda\mathbf{1}=0$ and $\mathbf{1}^\mathsf{T}w=1$. Since $M$ is positive definite, it is invertible, and the first equation gives $w=(\lambda/2)M^{-1}\mathbf{1}$. Applying the constraint yields \[ 1=(\lambda/2)\mathbf{1}^\mathsf{T}M^{-1}\mathbf{1}, \] so $\lambda/2=1/(\mathbf{1}^\mathsf{T}M^{-1}\mathbf{1})$, which gives the displayed weights. The denominator is positive because $M^{-1}$ is positive definite and $\mathbf{1}\ne 0$. Finally, \[ R(w^*)=\frac{\mathbf{1}^\mathsf{T}M^{-1}MM^{-1}\mathbf{1}} {(\mathbf{1}^\mathsf{T}M^{-1}\mathbf{1})^2} =\frac{1}{\mathbf{1}^\mathsf{T}M^{-1}\mathbf{1}}. \] For any other feasible $w=w^*+v$, we have $\mathbf{1}^\mathsf{T}v=0$. The stationary equation implies $v^\mathsf{T}Mw^*=(\lambda/2)v^\mathsf{T}\mathbf{1}=0$, and hence \[ R(w)-R(w^*)=2v^\mathsf{T}Mw^*+v^\mathsf{T}Mv=v^\mathsf{T}Mv>0 \] whenever $v\ne 0$, by positive definiteness. Thus the minimizer is unique. $\square$

Positive definiteness is a convenient sufficient condition, but it is stronger than necessary for uniqueness under the sum-to-one constraint. The relevant directions are only those that preserve the sum of the weights.

Uniqueness criterion: For a positive semidefinite error second-moment matrix $M$, the constrained risk has a unique minimizer if and only if $$ v^\mathsf{T}Mv>0 \quad\text{for every nonzero }v\text{ such that }\mathbf{1}^\mathsf{T}v=0. $$ Equivalently, the only vector in $\operatorname{null}(M)$ whose entries sum to zero must be the zero vector.

To verify the criterion, consider any feasible $w$ and any feasible perturbation $v$, so $\mathbf{1}^\mathsf{T}v=0$. If $v^\mathsf{T}Mv=0$, positive semidefiniteness implies $Mv=0$: for example, write $M=Q\Lambda Q^\mathsf{T}$ with nonnegative eigenvalues, so $v^\mathsf{T}Mv=0$ forces every positive-eigenvalue component of $v$ to vanish. Then $R(w+v)=R(w)$ for every $w$. Conversely, if there is no nonzero feasible direction with zero quadratic form, the risk is strictly convex along the feasible affine set, so it can have at most one minimizer.

A minimizer exists even when $M$ is singular. To see this, factor $M=A^\mathsf{T}A$, which is possible for every positive semidefinite matrix. Then $R(w)=\lVert Aw\rVert^2$. The set of values $Aw$ for which $\mathbf{1}^\mathsf{T}w=1$ is an affine subspace of a finite-dimensional space, hence is closed. A point in a nonempty closed affine subspace has a closest point to the origin, so this squared norm attains its minimum. The uniqueness criterion therefore distinguishes a unique minimizer from multiple minimizers, rather than from nonattainment.

Worked Example: Correlated Errors Change the Best Combination

Suppose both member errors have mean $0.1$, variance $1$, and covariance $0.8$. Then

C=\begin{pmatrix}1&0.8\\0.8&1\end{pmatrix}, \qquad bb^\mathsf{T}=\begin{pmatrix}0.01&0.01\\0.01&0.01\end{pmatrix}, \qquad M=\begin{pmatrix}1.01&0.81\\0.81&1.01\end{pmatrix}.

The two members have identical error moments, so swapping their weights leaves the risk unchanged. With $w_1+w_2=1$, the symmetric candidate is $(1/2,1/2)$; the two-by-two formula gives \[ w_1=\frac{1.01-0.81}{1.01+1.01-2(0.81)} =\frac{0.20}{0.40}=\frac12, \qquad w_2=\frac12. \] Its risk is \[ R(w)=\frac14(1.01)+\frac12(0.81)+\frac14(1.01)=0.91. \] The high covariance limits the benefit of averaging: if the errors had been independent with the same marginal variances and biases, the off-diagonal entries of $M$ would instead be $0.01$, and the equal-weight risk would be $0.51$.

Worked Example: A Singular Matrix Can Still Give a Unique Optimum

Let $e_1$ have mean zero and variance $1$, and let $e_2=0$ with probability one. Then \[ C=M=\begin{pmatrix}1&0\\0&0\end{pmatrix}, \] which is singular. For feasible weights $(w_1,1-w_1)$, the risk is \[ R(w)=w^\mathsf{T}Mw=w_1^2. \] This is minimized uniquely at $w_1=0,\ w_2=1$, with risk $0$. The feasible perturbations have the form $v=(t,-t)^\mathsf{T}$, and for $t\ne0$, \[ v^\mathsf{T}Mv=t^2>0. \] Thus the uniqueness criterion holds even though $M$ is not invertible. The inverse-based formula is unavailable here, but the constrained problem itself has a unique solution.

Common Pitfall and Practical Interpretation

A common mistake is to minimize $w^\mathsf{T}Cw$ and call the result the minimum squared-error ensemble. That expression is only the variance of the combined error. The full risk is $w^\mathsf{T}Cw+(w^\mathsf{T}b)^2$. For instance, an ensemble may reduce variance while assigning weights that make its combined bias large; omitting the second term can therefore select the wrong weights.

Another important qualification is that the sum-to-one constraint is what lets us express the ensemble error solely through the member errors $e_j=f_j(X)-Y$. Without it, the prediction error is $\sum_j w_j e_j+(\mathbf{1}^\mathsf{T}w-1)Y$, which also depends on the response $Y$. The error covariance matrix and bias vector alone then do not generally determine the risk. Here the weights have no sign restrictions, but they must still sum to one. A covariance matrix that is singular also does not automatically imply nonuniqueness: only a zero-risk-curvature direction that preserves the sum can create multiple optima.

In practice, the formula describes an oracle choice when the population error moments are known. It shows precisely how bias and dependence enter the target weights; estimating those moments from data is a separate problem. The main diagnostic is whether the risk has curvature in every feasible direction. If it does, the optimum is unique; if not, distinct weight vectors can achieve the same risk.

Check Your Understanding

Use the definitions and results above to answer the following questions.

Why does the condition $\mathbf{1}^\mathsf{T}w=1$ allow the ensemble error to be written as $w^\mathsf{T}e$?
Given an error covariance matrix $C$ and bias vector $b$, what is the error second-moment matrix $M$, and why is it positive semidefinite?
State the optimal-weight formula when $M$ is positive definite. What condition must its denominator satisfy?
For a singular positive semidefinite $M$, which feasible directions determine whether the minimizer is unique?
Why can minimizing only $w^\mathsf{T}Cw$ give weights that do not minimize squared-error risk?

Optimal Linear Weights From the Prediction-Error Covariance

What You'll Learn