Randomized Gibbs Predictions and Deterministic Mixtures

Two Ways to Use Gibbs Weights

A Gibbs distribution over candidate predictors can be used in two different ways. One can draw a candidate at random and use that candidate's prediction, or average the candidates' predictions and use the average. These procedures are not the same: the first randomizes over whole candidate outputs, while the second forms a new prediction. Their risks are related, but a risk comparison requires conditions on the loss.

The weights may come from empirical losses and a temperature parameter, but the comparison below needs only a finite set of candidates and fixed weights. If the weights were estimated from training data, all statements can be read conditionally on that training data. This makes clear that the risk comparison concerns a fresh test observation, not the data used to construct the weights.

For example, Gibbs weights constructed from empirical candidate losses \(\widehat R_j\), a prior \(\pi_j\), and temperature \(\eta>0\) have the form \(w_j=\pi_j\exp(-\eta\widehat R_j)/\sum_{k=1}^M\pi_k\exp(-\eta\widehat R_k)\). Once the weights are specified, the identities and comparison in this tutorial do not depend on how they were obtained.

Risk of a Randomized Predictor

Let \(Z=(X,Y)\) be a fresh test observation, and let \(\ell(y,a)\) be the loss incurred when the response is \(y\) and the prediction is \(a\). Define the risk of candidate \(f_j\) by \(R(f_j)=E[\ell(Y,f_j(X))]\). Suppose the candidate losses are integrable, so these expectations exist and are finite. The random index \(J\) is drawn independently of \(Z\), conditional on any training data used to set the weights.

The identity follows by conditioning on the finite-valued index \(J\). In particular, randomization averages the candidates' losses, not their predictions:

The second equality uses the conditional independence of \(J\) and the fresh test observation. If \(J\) is drawn once and the selected predictor is then used for many test observations, the same identity holds for the expected per-observation risk. The resulting losses across multiple observations need not be independent, since the same random choice is shared.

When Averaging Predictions Helps

To compare the randomized predictor with the deterministic mixture, the prediction values must be averageable. Suppose, for each input \(x\), the predictions lie in a convex subset \(C\) of a real vector space, and the loss \(a\mapsto\ell(y,a)\) is convex on \(C\) for every response \(y\). Convexity means that for any \(a,b\in C\) and \(\lambda\in[0,1]\), \(\ell(y,\lambda a+(1-\lambda)b)\leq\lambda\ell(y,a)+(1-\lambda)\ell(y,b)\). Also assume the mixture loss is integrable. These conditions ensure that \(\bar f(x)\) is an admissible prediction and that its risk is defined.

For completeness, the finite-weight inequality needed here follows directly from the definition of convexity. For fixed \(y\), we show \(\ell(y,\sum_{j=1}^M w_ja_j)\leq\sum_{j=1}^M w_j\ell(y,a_j)\) for \(a_j\in C\). The claim is immediate for one positive weight. For \(M\) positive weights, let \(s=\sum_{j=1}^{M-1}w_j\). If \(s=0\), the mixture is \(a_M\) and equality holds. If \(s>0\), set \(b=\sum_{j=1}^{M-1}(w_j/s)a_j\), which lies in \(C\). Applying convexity to \(sb+w_Ma_M\), and then applying the same claim to the first \(M-1\) normalized weights, gives

Weights equal to zero can be discarded, so the argument covers all nonnegative weights summing to one. Apply this inequality with \(a_j=f_j(X)\) and \(y=Y\). Taking expectations preserves the inequality, and the randomized-risk identity then yields the result:

This argument is conditional on the training data if the weights and candidates depend on that data. Averaging the resulting conditional inequality over the training data gives the corresponding unconditional comparison, whenever the risks are integrable. \(\square\)

What the Comparison Does—and Does Not—Say

The exact randomized-risk identity is broader than the deterministic-mixture guarantee. For a nonconvex loss, averaging predictions can increase loss, and a weighted average may not even be a meaningful prediction. The convexity condition concerns the loss as a function of the prediction, with the response held fixed. Convexity in some other argument does not establish the theorem.

Hard classification gives a useful warning. Suppose two classifiers predict labels \(1\) and \(0\), with weights \(0.6\) and \(0.4\), and the true label is \(0\). Under zero-one loss, randomized prediction has risk \(0.6(1)+0.4(0)=0.6\). A weighted vote chooses label \(1\), so its zero-one loss is \(1\), which is worse. The set of hard labels \(\{0,1\}\) is not convex as a prediction space, and zero-one loss as a function of a hard label does not provide the required convexity comparison. Averaging class scores or probabilities and then applying a decision rule is a different procedure; the theorem applies only if the resulting prediction and loss meet its stated conditions.

A related distinction is between averaging predictions and averaging risks. Randomized prediction averages the candidate risks exactly. The deterministic mixture first averages the prediction values, then evaluates the loss. Convexity is what permits the loss of that average to be bounded by the average loss; without it, the two operations cannot generally be interchanged.