Duplicate Predictors and Prior-Mass Sensitivity

Why Duplicate Candidates Matter

A prediction dictionary is often described as a collection of candidate rules, but an algorithm may operate on an indexed list. Those are not always the same thing: an indexed list can contain the same predictor more than once. Under a uniform prior, each index receives equal mass, so duplicating one predictor increases the total mass assigned to it. The Gibbs aggregate can then change even though the set of distinct predictions has not.

The key distinction is between adding an index with its own uniform-prior mass and splitting an existing candidate’s prior mass among duplicate indices. The first changes the prior over distinct predictions; the second does not. This distinction also clarifies which risk comparisons are stable: a prior-sensitive guarantee depends on the mass assigned to its comparator, not just on the number of distinct predictions.

Indexed Dictionaries and Gibbs Weights

Let \(f_1,\ldots,f_M\) be predictors indexed by \(j\). Given observations \((X_i,Y_i)\), define the empirical risk of candidate \(j\) by

\widehat R_j=\frac{1}{n}\sum_{i=1}^n \ell\bigl(f_j(X_i),Y_i\bigr).

Here \(\ell\) is a loss function, and assume the displayed empirical risks are finite. For a prior probability vector \(\pi=(\pi_1,\ldots,\pi_M)\) with \(\pi_j>0\), and temperature parameter \(\eta>0\), the Gibbs weight of index \(j\) is

w_j=\frac{\pi_j\exp(-\eta n\widehat R_j)} {\sum_{k=1}^M\pi_k\exp(-\eta n\widehat R_k)}.

The denominator normalizes the weights to sum to one. A deterministic mixture predicts \(\sum_j w_j f_j(x)\) when predictions are numerical and the aggregation operation is averaging. A randomized Gibbs prediction instead draws an index with probabilities \(w_j\) and uses that candidate. If several indices represent the same predictor, their weights must be added to find the total weight of that prediction.

Definition: A group \(G\) of indices is a group of exact duplicates if \(f_j(x)=f_k(x)\) for every input \(x\) and every \(j,k\in G\). Its prior mass is \(\pi(G)=\sum_{j\in G}\pi_j\), and its posterior Gibbs mass is \(w(G)=\sum_{j\in G}w_j\). Exact duplication concerns the predictors as functions, not merely their observed losses on one data set.

Worked Example: Uniform Duplication Changes the Aggregate

Suppose there are two numerical predictors, \(A\) and \(B\), with predictions \(f_A(x)=0\) and \(f_B(x)=1\) for every \(x\). Their empirical risks are \(0.2\) and \(0.5\). Take \(\eta n=2\), and first assign a uniform prior to the two indices.

The unnormalized weights are \(\exp(-0.4)\approx0.6703\) for \(A\) and \(\exp(-1)\approx0.3679\) for \(B\). Thus

w_A=\frac{0.6703}{0.6703+0.3679}\approx0.6457, \qquad w_B\approx0.3543.

The deterministic mixture predicts \(0.6457(0)+0.3543(1)=0.3543\). Now list \(A\) twice, keeping one index for \(B\), and again use a uniform prior on the expanded list. Each index has prior mass \(1/3\). The combined unnormalized mass for the two copies of \(A\) is \(2(0.6703)/3\), while that for \(B\) is \(0.3679/3\). Hence

w(\{A_1,A_2\})= \frac{2(0.6703)}{2(0.6703)+0.3679}\approx0.7848, \qquad w_B\approx0.2152.

The new mixture predicts \(0.2152\), not \(0.3543\). The loss attached to \(A\) did not change; the change comes from giving the duplicated prediction \(2/3\) of the prior mass instead of \(1/2\).

Equivalent Prior Splits Preserve Group Weight

A different operation is to retain the original total prior mass of a predictor while dividing that mass among its copies. This creates more indices, but it does not change the prior over distinct predictors. The following identity explains why the Gibbs aggregate is then unchanged.

Proposition: Suppose a candidate \(f\) with prior mass \(a>0\) and empirical risk \(r\) is replaced by \(m\) exact copies with prior masses \(a_1,\ldots,a_m>0\), where \(\sum_{s=1}^m a_s=a\). All other candidates and prior masses remain unchanged. Then the total Gibbs weight of the copies equals the original Gibbs weight of \(f\). The relative Gibbs weights among the copies are \(a_s/a\).

Proof. Write \(Z_0\) for the sum of the unnormalized Gibbs weights of all candidates other than \(f\). Before the split, the unnormalized weight of \(f\) is \(a\exp(-\eta n r)\), so its normalized weight is

\frac{a\exp(-\eta n r)} {Z_0+a\exp(-\eta n r)}.

After the split, the copies have the same empirical risk \(r\). Their combined unnormalized weight is

\sum_{s=1}^m a_s\exp(-\eta n r) =a\exp(-\eta n r).

The full normalizing constant is therefore still \(Z_0+a\exp(-\eta n r)\), proving that their combined normalized weight equals the original weight. The weight of copy \(s\), divided by the combined weight of all copies, is \(a_s/a\). Since every copy makes the same prediction, replacing the original weight by these divided weights leaves both the randomized prediction distribution over distinct predictors and the deterministic mixture unchanged. \(\square\)

Worked Example: Splitting Mass Without Changing the Prediction

Return to predictors \(A\) and \(B\), with risks \(0.2\) and \(0.5\), and \(\eta n=2\). Give \(A\) prior mass \(0.4\) and \(B\) prior mass \(0.6\). The unnormalized masses are

0.4e^{-0.4}\approx0.2681, \qquad 0.6e^{-1}\approx0.2207.

The total normalizer is approximately \(0.4888\), so \(A\)'s Gibbs mass is \(0.2681/0.4888\approx0.5486\), and \(B\)'s is approximately \(0.4514\). Now replace \(A\) by two exact copies with prior masses \(0.1\) and \(0.3\). Their unnormalized masses are \(0.1e^{-0.4}\approx0.0670\) and \(0.3e^{-0.4}\approx0.2011\). They sum to \(0.2681\), so the normalizer and total mass on \(A\) are unchanged. The copies receive one-quarter and three-quarters of \(A\)'s total Gibbs mass, respectively, but both predict zero. The deterministic aggregate remains \(0.4514\).

By contrast, assigning prior mass \(1/3\) to each index after making the copy would give \(A\) total prior mass \(2/3\), rather than \(0.4\). That is not a split of the original prior; it is a different prior.

Prior Mass and an Oracle Inequality

The dependence of a Gibbs comparison on prior mass can be seen directly in a finite-dictionary inequality. Let \(\widehat R_j\) be finite, let \(\pi_j>0\) sum to one, and define \(w_j\) by the Gibbs formula above. Set

Z=\sum_{j=1}^M\pi_j\exp(-\eta n\widehat R_j).

For probability vectors \(w\) and \(\pi\), define \(\operatorname{KL}(w\|\pi)=\sum_j w_j\log(w_j/\pi_j)\). In this setting each Gibbs weight is positive, so this expression is well-defined.

Theorem: For each index \(j\),

\sum_{k=1}^M w_k\widehat R_k \leq \widehat R_j+\frac{\log(1/\pi_j)}{\eta n}.

If a group \(G\) has the same empirical risk \(\widehat R_G\) for every index in it, then

\sum_{k=1}^M w_k\widehat R_k \leq \widehat R_G+\frac{\log(1/\pi(G))}{\eta n}.

Proof. The Gibbs definition gives \(\log(w_k/\pi_k)=-\eta n\widehat R_k-\log Z\). Multiplying by \(w_k\) and summing yields

\operatorname{KL}(w\|\pi) =-\eta n\sum_{k=1}^M w_k\widehat R_k-\log Z.

Rearranging and using \(\operatorname{KL}(w\|\pi)\geq0\) gives

\sum_{k=1}^M w_k\widehat R_k =-\frac{\log Z}{\eta n} -\frac{\operatorname{KL}(w\|\pi)}{\eta n} \leq-\frac{\log Z}{\eta n}.

For any index \(j\), \(Z\geq\pi_j\exp(-\eta n\widehat R_j)\). Taking negative logarithms and dividing by \(\eta n>0\) proves the first inequality. If all indices in \(G\) have risk \(\widehat R_G\), then their terms in \(Z\) sum to \(\pi(G)\exp(-\eta n\widehat R_G)\). Thus \(Z\geq\pi(G)\exp(-\eta n\widehat R_G)\), which proves the group inequality by the same steps. \(\square\)

The group form matters because a guarantee for one index and a guarantee for a distinct prediction need not use the same prior mass. If a predictor's mass \(a\) is divided among exact copies, an individual copy may have mass \(a_s\), but the group retains mass \(a\). Comparing with that prediction as a group keeps the complexity term \(\log(1/a)/(\eta n)\), rather than replacing it with the larger single-copy term \(\log(1/a_s)/(\eta n)\).

Worked Example: Which Oracle Penalty Changes?

There are two distinct predictors, \(A\) and \(B\), each with empirical risk \(0.2\) and \(0.4\), respectively, and \(\eta n=10\). Under the uniform prior on the two predictors, each has prior mass \(1/2\). The inequality for \(A\) gives an upper bound of

0.2+\frac{\log 2}{10}\approx0.2693

on the Gibbs-weighted empirical risk. Now make five exact copies of \(A\), retain one index for \(B\), and use a uniform prior on all six indices. The total prior mass of the \(A\) group is \(5/6\). Applying the group inequality gives

0.2+\frac{\log(6/5)}{10}\approx0.2182.

The uniform expanded prior puts more mass on \(A\), so its group penalty is smaller; it also changes the Gibbs aggregate. A bound for the single \(B\) index uses mass \(1/6\), giving \(0.4+\log(6)/10\approx0.5792\), rather than the original \(0.4+\log(2)/10\approx0.4693\). If instead the original mass \(1/2\) on \(A\) were split equally among its five copies, each would receive \(1/10\), while \(B\) would retain mass \(1/2\). The aggregate and the group penalty for \(A\) would then be unchanged.

What Is and Is Not Invariant

For exact duplicates, the minimum empirical risk among distinct predictions is unchanged: adding another copy cannot create a new prediction or lower that minimum. If a prior is split while preserving every distinct predictor's total mass, the Gibbs weight assigned to each distinct predictor is unchanged. Consequently, any deterministic mixture formed from those group weights is the same function. Its empirical risk is unchanged, and its population risk is unchanged whenever that risk is defined for the same prediction rule and data distribution.

A uniform prior on an expanded list is different. It changes the group masses to \(m_G/M\), where \(m_G\) is the number of indices in group \(G\). The Gibbs aggregate can change, and the prior-mass term in an oracle inequality changes with it. A risk guarantee that depends on the resulting weights or on prior mass is therefore not generally invariant to uniform duplication. Nor does the fact that a candidate is duplicated imply any improvement in the intrinsic quality of the set of distinct predictions.

There is a further qualification: two candidates with identical losses on the observed sample need not be exact duplicates as functions. Their Gibbs weights may combine in the same way on that sample, but their predictions can differ at new inputs. In that case, equality of empirical losses alone does not establish equality of the aggregate prediction or of population risk.

Common Mistake

Wrong reasoning: “The list of distinct predictors is unchanged, so a uniform-prior Gibbs aggregate and every associated guarantee must be unchanged.”
Correction: A uniform prior is uniform over indices, not necessarily over distinct predictions. Duplicating a predictor increases its group mass and can change the aggregate. To preserve the original aggregate, divide the original prior mass among the copies so their total mass stays fixed. For oracle comparisons, use the total prior mass of an exact-duplicate group when the comparator is the shared prediction.

Key takeaway: Duplication is harmless only when total prior mass on each distinct prediction is preserved. Uniform duplication changes that mass; a prior split does not. The relevant complexity term is determined by the mass assigned to the comparator—or to its duplicate group—not merely by the number of distinct predictors.

Check Your Understanding

Answer each question by distinguishing index-level mass from mass on distinct predictions.

A predictor with prior mass \(0.3\) is replaced by three exact copies with prior masses \(0.05\), \(0.10\), and \(0.15\). What happens to its total Gibbs mass, and why?
Under a uniform prior, one of four candidate indices is duplicated once. What are the prior masses of the duplicated prediction's group and each remaining singleton?
Why can two candidates with identical empirical risks still produce different population risks after aggregation?
In the group oracle inequality, why does a duplicate group use \(\pi(G)\) rather than the prior mass of one selected copy?
State one risk quantity that remains unchanged under a mass-preserving split of exact duplicates, and one feature that can change under uniform duplication.