Selecting Subensembles Under Correlated Candidate Errors

Why Subset Selection Depends on Joint Errors

A dictionary of candidate predictors can contain many useful models, but selecting a subset is not simply a matter of keeping the predictors with the smallest individual errors. When errors move together, averaging them may leave much of their error intact; when errors offset one another, a group of individually imperfect predictors may average well. The relevant object is therefore the joint behavior of the errors in each proposed subset.

This tutorial studies fixed subsets combined with equal weights under squared prediction loss. The risk comparisons are population comparisons for a fresh test observation. They describe the risk of a specified subset; if the subset is itself chosen from data, the risk of the resulting selection procedure also includes the effects of that selection.

The errors are defined against the same response \(Y\), so their dependence is part of the prediction problem, not an optional adjustment. Define the error second-moment matrix \(Q\) by \(Q_{j\ell}=E[e_j e_\ell]\). This is not necessarily a covariance matrix: if the candidates have nonzero mean errors, then the second moments include both their covariance and their biases.

The double sum includes the diagonal terms \(Q_{jj}=E[e_j^2]\), which are the candidates' individual risks, and the off-diagonal terms \(Q_{j\ell}\), which describe how their errors co-occur. For a fixed subset size, the denominator \(k^2\) is the same across subsets. Thus the best subset of that size minimizes the sum of all its diagonal and off-diagonal second moments. A correlation is only a standardized off-diagonal quantity; it does not by itself reveal the size of the cross-moment or the diagonal risks.

A Risk Formula for Exchangeable Errors

A useful benchmark is a dictionary whose errors have a common mean, common variance, and common pairwise correlation. This symmetry makes the role of subset size explicit. Write \(b=E[e_j]\), \(\sigma^2=\operatorname{Var}(e_j)>0\), and suppose \(\operatorname{Corr}(e_j,e_\ell)=\rho\) for every distinct pair. For a dictionary of size \(M\geq2\), such an equicorrelation matrix must satisfy \(-1/(M-1)\leq\rho\leq1\).

Proof. The mean error of the average is \(b\). Each error has variance \(\sigma^2\), and each distinct pair has covariance \(\rho\sigma^2\). Expanding the variance of the average gives

For any random variable \(A\) with finite second moment, \(E[A^2]=\operatorname{Var}(A)+(E[A])^2\). Applying this identity to the average error adds \(b^2\) to the displayed variance and proves the formula. Since it depends on \(S\) only through \(k\), every subset of that size has the same risk. \(\square\)

For \(\rho<1\), increasing \(k\) decreases the term \((1-\rho)\sigma^2/k\), but the risk approaches \(b^2+\rho\sigma^2\), not necessarily zero. The shared component remains in the average. At \(\rho=1\), all errors are perfectly correlated and averaging does not reduce their variance. If \(\rho\) is negative, the formula still applies within the stated positive-semidefinite range; offsetting errors can make the risk decrease more quickly with subset size.

Selection Within a Fixed Subset Size

When candidate errors are not exchangeable, the risk depends on which members are chosen, not just on how many. For a subset \(S\) with size \(k\), define its total second-moment sum as \(A(S)=\sum_{j\in S}Q_{jj}+2\sum_{j<\ell,\ j,\ell\in S}Q_{j\ell}\). Symmetry of \(Q\) gives \(R(S)=A(S)/k^2\). This provides an exact population criterion for comparisons among subsets of equal size.

This criterion explains why there is no universal ranking of candidates that can be used independently of the subset. A candidate with a relatively large individual risk may still help a particular group if its errors have sufficiently negative cross-moments with that group's errors. Conversely, several strong individual predictors can make a poor subset if their errors are strongly aligned.

Interpretation and a Common Pitfall

The risk matrix makes dependence concrete: for each proposed subset, assess both the diagonal risks and the pairwise cross-moments. Under common, zero-mean errors, equal variances, and common pairwise correlation, a single correlation parameter is enough to describe the risk by subset size. Outside that symmetric setting, candidate-specific risks, biases, and dependencies all matter. Correlation summaries can be useful descriptions, but they are not substitutes for the risk calculation.

Common mistake: choosing the lowest-risk candidates and assuming their average must be best. This ignores the off-diagonal terms. In the worked example, candidate 3 is individually best, yet the pair \(\{1,2\}\) has lower risk than either pair containing candidate 3 because its negative cross-moment reduces the average's error. The correction is to compare the full subset risk, including all cross-moments, rather than rank candidates by their individual risks alone.

There is also an important distinction between an oracle comparison and a data-driven selection procedure. The formulas above compare specified subsets using their population error moments. If a subset is chosen after inspecting data, the selected subset is random, and evaluating that selection procedure requires accounting for how the choice was made. The fixed-subset identity by itself does not remove that selection effect.