Paired Risk Comparisons for Competing Ensembles

Why Compare Risks in Pairs?

When two ensembles are evaluated on the same test observations, their errors are naturally paired: each observation supplies one loss for ensemble A and one for ensemble B. The difference between those losses, observation by observation, is often more informative than treating the two lists of losses as unrelated. If an observation is difficult for both ensembles, that shared difficulty can contribute little to their difference.

The goal here is to quantify uncertainty in the difference between two risks. A negative estimated difference will favor A under the convention below; a positive one will favor B. The size of the uncertainty depends not only on how variable each ensemble’s losses are, but also on how the two losses co-vary on the same observations.

Risk Difference and Paired Data

Let \(Z=(X,Y)\) denote a new observation from the evaluation population, and let \(\hat f_A\) and \(\hat f_B\) be two fitted ensembles. For a specified loss function \(\ell\), define their losses on \(Z\) by \(L_A=\ell(Y,\hat f_A(X))\) and \(L_B=\ell(Y,\hat f_B(X))\). The fitted ensembles are treated as fixed for the comparison. If they were trained on random data independent of the evaluation sample, these definitions and results can instead be read conditionally on the training data.

The pairing is part of the data structure: \(L_{Ai}\) and \(L_{Bi}\) come from the same \(Z_i\). The risk difference concerns their expected difference, not the difference between losses from two independently sampled evaluation sets.

Exact Variance of the Paired Risk Difference

Proof. Since the observations are identically distributed, each \(D_i=L_{Ai}-L_{Bi}\) has expectation \(E[D_i]=R_A-R_B=\Delta\). Linearity of expectation gives \(E[\hat\Delta]=n^{-1}\sum_{i=1}^n E[D_i]=\Delta\). Independence of the evaluation observations makes the \(D_i\) independent, so \(\operatorname{Var}(\hat\Delta)=n^{-2}\sum_{i=1}^n\operatorname{Var}(D_i)=\operatorname{Var}(D_1)/n\). Finally, expanding the variance of a difference gives \(\operatorname{Var}(L_A-L_B)=\operatorname{Var}(L_A)+\operatorname{Var}(L_B)-2\operatorname{Cov}(L_A,L_B)\). The finite second-moment assumptions ensure these variances and covariances exist. \(\square\)

This identity explains the role of pairing. Positive covariance reduces the variance of the difference relative to the sum of the two marginal variances; negative covariance increases it. The comparison is not automatically precise just because the ensembles share some predictions or training data: what matters directly is the covariance of their losses on the evaluation distribution.

Estimating Uncertainty From the Paired Differences

The population variance of \(D_i\) is generally unknown. Compute the observed differences \(d_i=L_{Ai}-L_{Bi}\), their mean \(\bar d=\hat\Delta\), and their sample variance:

The estimated standard error of the paired risk difference is \(s_D/\sqrt{n}\). This estimate uses the empirical variability of the differences themselves, so it automatically incorporates the observed association between A’s and B’s losses.

To verify the unbiasedness, write \(\mu_D=E[D_i]\) and \(\sigma_D^2=\operatorname{Var}(D_i)\). The identity \(\sum_{i=1}^n(D_i-\bar D)^2=\sum_{i=1}^n(D_i-\mu_D)^2-n(\bar D-\mu_D)^2\) implies, on taking expectations, that the left side has expectation \(n\sigma_D^2-n(\sigma_D^2/n)=(n-1)\sigma_D^2\). Dividing by \(n-1\) proves \(E[s_D^2]=\sigma_D^2\). Since \(n\) is fixed, \(E[s_D^2/n]=\sigma_D^2/n=\operatorname{Var}(\hat\Delta)\).

For large \(n\), if the paired differences are identically distributed with finite, nonzero variance, the central limit theorem and Slutsky’s theorem give \((\hat\Delta-\Delta)/(s_D/\sqrt n)\xrightarrow{d}N(0,1)\). This motivates an approximate confidence interval \(\hat\Delta\pm z_{1-\alpha/2}s_D/\sqrt n\), where \(z_{1-\alpha/2}\) is the corresponding standard normal quantile. This is an asymptotic interval, not an exact finite-sample guarantee. In particular, paired differences need not be normally distributed merely because the two ensemble losses have simple marginal distributions.

Shared Loss Components and Ranking Precision

A simple model shows how shared losses can cancel. Suppose a paired loss can be written as \(L_A=C+U\) and \(L_B=C+V\), where \(C\) is a common component and \(U,V\) are ensemble-specific components. If \(C,U,V\) are mutually independent and have finite variances, then \(\operatorname{Var}(L_A-L_B)=\operatorname{Var}(U-V)=\operatorname{Var}(U)+\operatorname{Var}(V)\). The common component contributes to each ensemble’s risk variability but disappears from their difference.

Common Mistake: Forgetting Which Losses Belong Together

A common mistake is to compare the two empirical risks as if they came from independent test samples, using a variance based only on \(\operatorname{Var}(L_A)+\operatorname{Var}(L_B)\). That discards the covariance term even when both ensembles were evaluated on the same observations. The correction is to calculate \(d_i=L_{Ai}-L_{Bi}\) for every observation first, then estimate uncertainty from the sample variance of those paired differences.

Pairing does not guarantee lower uncertainty: it helps when the paired losses have positive covariance and can hurt when that covariance is negative. Nor does pairing fix dependence between evaluation observations. The formulas above assume an independent evaluation sample; repeated or clustered observations require uncertainty calculations that account for their dependence. If ensemble fitting is random, the stated variance describes evaluation-sample uncertainty conditional on the fitted ensembles. Variation across training sets is a separate source of uncertainty, summarized by the law of total variance when the target requires it.