From Gibbs Risk to Deterministic Predictors

Why Average Predictions?

A posterior distribution over predictors describes more than one candidate rule: it also defines a randomized prediction procedure, often called a Gibbs predictor. In some applications, however, we want one deterministic prediction at each input. If the predictions can be averaged and the loss is convex in the prediction, Jensen’s inequality gives a direct way to compare that deterministic rule with the randomized one. This comparison is useful because a risk guarantee for the Gibbs predictor can then yield a risk guarantee for the averaged predictor.

The conversion depends on the geometry of the prediction space and on convexity of the loss in its prediction argument. It is not a universal property of posterior averaging. In particular, averaging hard class labels and then applying zero-one loss is not the same operation as averaging predictions under a convex loss.

Gibbs Risk and the Posterior-Averaged Predictor

Let $(X,Y)$ have a fixed joint distribution $P$. A predictor $h$ maps an input $x$ to an action $h(x)$ in a convex subset $C$ of a real vector space, such as $\mathbb{R}$ or $\mathbb{R}^d$. Let $\ell(a,y)$ be the loss for taking action $a$ when the outcome is $y$. Assume the relevant maps are measurable. A posterior $\rho$ is a probability distribution on the predictor class; in this tutorial, it is held fixed when defining the risks below.

Definition: The risk of a predictor $h$ is $R(h)=E[\ell(h(X),Y)]$. The Gibbs risk under $\rho$ is $$ R_G(\rho)=\int R(h)\,\rho(dh) =E\!\left[\int \ell(h(X),Y)\,\rho(dh)\right]. $$ When the posterior mean exists, the posterior-averaged predictor is $$ \bar h(x)=\int h(x)\,\rho(dh). $$ The deterministic risk is $R(\bar h)=E[\ell(\bar h(X),Y)]$.

The posterior mean must be a valid action: the integral defining $\bar h(x)$ must exist and lie in $C$, at least for $P_X$-almost every $x$. We also require $\bar h$ to be measurable. For nonnegative losses, the risks may be infinite; for the comparisons involving finite differences below, we will state stronger integrability assumptions. The expression for $R_G$ describes drawing $h$ from $\rho$ and evaluating its loss, then averaging over both the data and the predictor draw.

Worked Example: Squared Error With Two Constant Predictors

Let $Y=1$ with probability one, let $X$ be constant, and let $\rho$ assign probability $1/2$ to each of the constant predictors $h_0(x)=0$ and $h_2(x)=2$. Under squared-error loss $\ell(a,y)=(a-y)^2$, the Gibbs risk is $$ R_G(\rho)=\frac12(0-1)^2+\frac12(2-1)^2 =\frac12+\frac12=1. $$

The posterior-averaged prediction is $\bar h(x)=\frac12(0)+\frac12(2)=1$, so its risk is $R(\bar h)=(1-1)^2=0$. In this case averaging strictly improves risk. The calculation is finite throughout, and the prediction space $\mathbb{R}$ is convex.

The Convex-Loss Conversion

The key assumption is that for each fixed outcome $y$, the function $a\mapsto \ell(a,y)$ is convex on $C$. Convexity means that for any $a_1,a_2\in C$ and $t\in[0,1]$, $\ell(ta_1+(1-t)a_2,y)\leq t\ell(a_1,y)+(1-t)\ell(a_2,y)$. Jensen’s inequality extends this comparison from finite averages to a probability distribution over predictors.

Theorem: Suppose $C$ is convex, $a\mapsto\ell(a,y)$ is convex for every $y$, and the posterior mean $\bar h(x)$ exists, is measurable, and belongs to $C$ for $P_X$-almost every $x$. Suppose also that, for $P$-almost every $(x,y)$, the posterior integral of the loss is well-defined and Jensen’s inequality applies (for example, the loss is nonnegative and the posterior mean exists). Then $$ R(\bar h)\leq R_G(\rho). $$ If signed losses are used, assume the required posterior and data expectations are integrable so that the displayed risks and integrations are well-defined.

Proof. Fix a pair $(x,y)$ for which the hypotheses hold. The predictions $h(x)$, under $h\sim\rho$, have posterior mean $\bar h(x)$. Convexity of the loss in its first argument and Jensen’s inequality give $$ \ell(\bar h(x),y) =\ell\!\left(\int h(x)\,\rho(dh),y\right) \leq \int \ell(h(x),y)\,\rho(dh). $$ Integrating this pointwise inequality over $(X,Y)$ preserves the inequality. For nonnegative losses, the exchange of the two integrals is justified by Tonelli’s theorem; under the stated integrability conditions it is also justified for signed losses. Therefore $$ R(\bar h)=E[\ell(\bar h(X),Y)] \leq E\!\left[\int \ell(h(X),Y)\,\rho(dh)\right] =R_G(\rho). $$ This proves the claim. $\square$

The inequality is deterministic for each posterior satisfying the assumptions. Thus, if a valid argument gives an upper bound on $R_G(\rho)$, the same upper bound also applies to $R(\bar h)$. This observation does not establish that a particular PAC-Bayes bound is valid; it only transfers a bound once the Gibbs-risk guarantee is available. The same pointwise reasoning applies to empirical averages over a fixed data set when the loss is convex.

Worked Example: Absolute Error and a Strict Improvement

Consider a single input and a fixed outcome $y=1$. Let the posterior put equal mass on predictions $h_-(x)=-2$ and $h_+(x)=2$. Under absolute-error loss, $\ell(a,y)=|a-y|$, the two losses are $$ |-2-1|=3,\qquad |2-1|=1. $$ Thus $R_G(\rho)=\frac12(3)+\frac12(1)=2$. The averaged prediction is $\bar h(x)=\frac12(-2)+\frac12(2)=0$, and its risk is $|0-1|=1$. Hence $R(\bar h)=1\leq 2=R_G(\rho)$. The loss is convex in the prediction, and the calculation exhibits a strict inequality.

Worked Example: Averaging Probabilities Under Log Loss

For a binary outcome $Y\in\{0,1\}$, let a predictor report a probability $p\in(0,1)$ for $Y=1$. The log loss at $y=1$ is $-\log p$, a convex function of $p$, since its second derivative is $1/p^2>0$. Suppose the observed outcome is $1$, and the posterior puts equal mass on predictions $p_1=0.2$ and $p_2=0.8$. The Gibbs loss is $$ R_G(\rho)=\frac12[-\log(0.2)]+\frac12[-\log(0.8)] =-\frac12\log(0.16)\approx 0.9163. $$ The averaged probability is $\bar p=(0.2+0.8)/2=0.5$, with deterministic log loss $$ -\log(0.5)=\log 2\approx 0.6931. $$ Therefore the averaged prediction has lower loss in this example. It remains a valid probability because $(0,1)$ is convex for these two predictions and their mean lies in $(0,1)$.

A Squared-Error Decomposition

For squared-error loss, one can say more than the inequality: the risk difference is exactly the average posterior variance of the prediction at the input. This identity requires finite risks; without that condition, subtracting risks that may both be infinite is not meaningful.

Theorem: Suppose $h(x)$ is real-valued, $\bar h(x)=\int h(x)\rho(dh)$ exists, and $$ E[Y^2]<\infty,\qquad E\!\left[\int h(X)^2\,\rho(dh)\right]<\infty. $$ For squared-error loss, define $\operatorname{Var}_{\rho}(h(x))=\int (h(x)-\bar h(x))^2\,\rho(dh)$. Then both risks are finite and $$ R_G(\rho)=R(\bar h)+E[\operatorname{Var}_{\rho}(h(X))]. $$ In particular, equality $R_G(\rho)=R(\bar h)$ holds if and only if $\operatorname{Var}_{\rho}(h(X))=0$ for $P_X$-almost every $x$.

Proof. First, Jensen’s inequality gives $E[\bar h(X)^2]\leq E[\int h(X)^2\rho(dh)]<\infty$. Also, $(a-b)^2\leq 2a^2+2b^2$, so the assumed second moments imply finite Gibbs squared-error risk and finite averaged-predictor squared-error risk. For fixed $x,y$, write $h(x)-y=(h(x)-\bar h(x))+(\bar h(x)-y)$. Squaring and integrating over $\rho$ gives $$ \begin{aligned} \int (h(x)-y)^2\,\rho(dh) &=\int (h(x)-\bar h(x))^2\,\rho(dh) +(\bar h(x)-y)^2\\ &\quad+2(\bar h(x)-y)\int(h(x)-\bar h(x))\,\rho(dh). \end{aligned} $$ The last integral is zero by the definition of $\bar h(x)$. Taking expectation over $(X,Y)$ yields the asserted identity. The variance term is nonnegative, so the risks are equal exactly when its expectation is zero. A nonnegative measurable function has expectation zero if and only if it is zero almost surely; applying this to $\operatorname{Var}_{\rho}(h(X))$ gives the stated condition. Finiteness proved at the start makes the subtraction and equality characterization valid. $\square$

Worked Example: Checking the Squared-Error Identity

Return to the two constant predictors $0$ and $2$, but now let $Y$ be $0$ or $2$, each with probability $1/2$. The posterior mean prediction is $1$. Its risk is $$ R(\bar h)=\frac12(1-0)^2+\frac12(1-2)^2 =\frac12+\frac12=1. $$ For either fixed posterior predictor, the squared error is $0$ on one outcome and $4$ on the other, so its risk is $2$. Consequently $R_G(\rho)=2$. At every input the posterior variance of the prediction is $$ \frac12(0-1)^2+\frac12(2-1)^2=1. $$ The decomposition checks: $R_G(\rho)=1+1=R(\bar h)+E[\operatorname{Var}_{\rho}(h(X))]$. Here $E[Y^2]=2$, and $E\int h(X)^2\rho(dh)=\frac12(0^2)+\frac12(2^2)=2$, so the finiteness assumptions hold.

When the Conversion Does Not Apply

Convexity must hold for the loss as a function of the action being averaged. It is not enough that the posterior itself is a probability distribution, or that a different surrogate loss is convex. The average must also be a valid prediction in the action space. These requirements matter especially for hard classification, where labels are discrete and zero-one loss is not convex in the label.

Worked Example: Majority Vote Can Have Higher Zero-One Risk

Take one input $x$, with true label $Y=0$ with probability one. Let the posterior assign probability $0.6$ to the constant classifier predicting $1$, and probability $0.4$ to the constant classifier predicting $0$. The Gibbs zero-one risk is $$ R_G(\rho)=0.6(1)+0.4(0)=0.6. $$ The posterior majority vote predicts $1$, so its deterministic zero-one risk is $R(\bar h_{\mathrm{vote}})=1$, which is greater than $0.6$. The posterior mean of the numerical codes $0$ and $1$ would be $0.6$, but that is not a hard class label. Thresholding that mean produces the wrong majority-vote label in this example. There is no contradiction with Jensen’s inequality: the hard-label action space is not convex, and zero-one loss does not provide the needed convexity in a valid hard-label action.

Common Pitfall

Assuming every posterior average improves risk. Jensen’s inequality requires convexity in the action being averaged, and the average must be a valid action. Majority vote under zero-one loss is not covered by the convex-loss argument.
Confusing a probability mixture with a hard decision. Averaging predicted probabilities can be a valid deterministic predictor under a convex probability loss. Converting those probabilities to a class label is a separate operation, and the convexity guarantee does not automatically survive it.
Using a squared-error identity with infinite risks. A formal variance decomposition is not enough if the two risks are infinite. For the finite-risk identity above, the second moments of both the outcome and posterior predictions were explicitly required.

Key takeaway: If the prediction space is convex, the posterior mean is well-defined and valid, and the loss is convex in the prediction, Jensen’s inequality gives $R(\bar h)\leq R_G(\rho)$. For squared error, finite second moments yield the sharper variance decomposition. Nonconvex losses and invalid averaged actions require a different argument.

Check Your Understanding

Answer each question by identifying the assumptions or calculation that matters.

State the two central conditions, beyond existence of the posterior, that allow Jensen’s inequality to compare averaged-predictor risk with Gibbs risk.
For squared-error loss, why is a condition such as $E[Y^2]<\infty$ useful when claiming a risk decomposition and an equality characterization?
A posterior puts equal mass on predictions $1$ and $5$, and the outcome is $3$. Compute the Gibbs squared-error risk, the averaged-predictor risk, and the posterior variance.
Explain why the majority-vote example does not contradict the convex-loss theorem.
When log loss is applied to an averaged predicted probability, which function is convex, and in which argument?