Gibbs Posteriors From Empirical Risk

Why a Gibbs Posterior?

A predictor with the smallest empirical risk may fit the observed sample well but ignore useful prior information or be unstable to small changes in that sample. PAC-Bayes methods express this trade-off by comparing a posterior distribution over predictors with a prior distribution. A Gibbs posterior makes the trade-off explicit: it gives greater weight to predictors with lower empirical risk, while the prior continues to affect which predictors receive weight.

The positive temperature controls the balance. A small temperature concentrates weight more strongly on low empirical risk; a large temperature keeps the posterior closer to the prior. The central result is not just a formula for weights: it says that the normalized Gibbs distribution is exactly the optimizer of empirical risk plus a temperature-scaled KL divergence.

Definition and Variational Characterization

Let $(\mathcal H,\mathcal A)$ be a measurable predictor space and let $\Pi$ be a probability measure on it, chosen independently of the training sample. For observed data $Z_1,\ldots,Z_n$, let $\widehat R_n(h)$ be the empirical risk of predictor $h$. Assume $\widehat R_n$ is measurable and bounded, and fix a temperature $\tau>0$. Boundedness ensures that the normalizing constant below is positive and finite.

Definition: The Gibbs posterior $Q_\tau$, with prior $\Pi$, empirical risk $\widehat R_n$, and temperature $\tau$, is the probability measure with density relative to $\Pi$ $$ \frac{dQ_\tau}{d\Pi}(h) = \frac{\exp\{-\widehat R_n(h)/\tau\}}{Z_\tau}, \qquad Z_\tau = \int_{\mathcal H}\exp\{-\widehat R_n(h)/\tau\}\,d\Pi(h). $$ The constant $Z_\tau$ is the normalizing constant, also called the partition function.

For probability measures $Q$ and $\Pi$, the KL divergence is $\mathrm{KL}(Q\|\Pi)=\int \log(dQ/d\Pi)\,dQ$ when $Q$ is absolutely continuous with respect to $\Pi$, and is $+\infty$ otherwise. Define the objective $\mathcal J_\tau(Q)=\int \widehat R_n(h)\,dQ(h)+\tau\mathrm{KL}(Q\|\Pi)$. A distribution that assigns mass where the prior assigns none has infinite KL divergence and cannot improve this objective.

Theorem: Under the conditions above, $Q_\tau$ uniquely minimizes $\mathcal J_\tau(Q)$, up to equality as probability measures. For every $Q$ with finite $\mathrm{KL}(Q\|\Pi)$, $$ \mathcal J_\tau(Q) = -\tau\log Z_\tau+\tau\mathrm{KL}(Q\|Q_\tau). $$ Consequently, $\inf_Q\mathcal J_\tau(Q)=-\tau\log Z_\tau$, attained at $Q=Q_\tau$.

Proof. The Gibbs density is strictly positive $\Pi$-almost everywhere because the empirical risk is bounded. Thus any $Q$ with finite $\mathrm{KL}(Q\|\Pi)$ is also absolutely continuous with respect to $Q_\tau$. Taking the logarithm of the density gives $\log(dQ_\tau/d\Pi)=-\widehat R_n/\tau-\log Z_\tau$. Therefore, $$ \mathrm{KL}(Q\|Q_\tau) = \mathrm{KL}(Q\|\Pi) +\frac{1}{\tau}\int \widehat R_n(h)\,dQ(h) +\log Z_\tau. $$ Multiplying by $\tau$ and rearranging proves the identity. KL divergence is nonnegative, with equality exactly when $Q=Q_\tau$. This proves both minimization and uniqueness. Distributions with infinite objective cannot be minimizers, so the result covers the full optimization problem. $\square$

This identity explains the role of normalization: $-\tau\log Z_\tau$ is the minimum value of the empirical-risk-plus-divergence objective. The partition function is not itself a generalization guarantee. Rather, it normalizes the posterior and encodes the optimized objective value.

Worked Examples

Worked Example: Two Predictors With Equal Prior Mass

Suppose the predictor space contains $h_1,h_2$, with prior probabilities $1/2,1/2$. Their empirical risks are $0.2$ and $0.4$, respectively, and $\tau=0.2$. Calculate the Gibbs probabilities, normalizer, posterior-averaged empirical risk, and KL divergence.

The unnormalized weights are $\frac12e^{-0.2/0.2}=\frac12e^{-1}$ and $\frac12e^{-0.4/0.2}=\frac12e^{-2}$. Hence $Z_\tau=\frac12(e^{-1}+e^{-2})\approx0.251607$. The normalized probabilities are $Q_\tau(h_1)=e/(e+1)\approx0.731059$ and $Q_\tau(h_2)=1/(e+1)\approx0.268941$.

The posterior-averaged empirical risk is $0.2(0.731059)+0.4(0.268941)\approx0.253788$. Using the variational identity at the minimizer, $$ \mathrm{KL}(Q_\tau\|\Pi) = -\frac{0.253788}{0.2}-\log(0.251607) \approx -1.268941+1.379886 \approx0.110945. $$ The positive divergence is the price of shifting mass away from the equal prior and toward the lower-risk predictor.

Worked Example: A Gaussian Prior and a Quadratic Empirical Risk

Let $\mathcal H=\mathbb R$, $\Pi=N(0,1)$, $\widehat R_n(h)=(h-1)^2$, and $\tau=1/2$. Find the posterior and check the variational identity. Here the quadratic risk is unbounded, but the defining integral is finite, so the Gibbs density is well-defined.

The posterior density is proportional to $\exp\{-h^2/2-2(h-1)^2\}$. Completing the square, $$ -\frac{h^2}{2}-2(h-1)^2 = -\frac{5}{2}(h-0.8)^2-0.4. $$ Thus $Q_\tau=N(0.8,0.2)$, and integrating the completed square gives $Z_\tau=e^{-0.4}/\sqrt{5}$, so $-\log Z_\tau=0.4+\frac12\log 5\approx1.204719$.

Under $Q_\tau$, the expected empirical risk is $E[(h-1)^2]=\operatorname{Var}(h)+(E[h]-1)^2=0.2+0.04=0.24$. For two normal distributions, $$ \mathrm{KL}\{N(0.8,0.2)\|N(0,1)\} = \frac12\{0.8^2+0.2-1-\log(0.2)\} \approx0.724719. $$ The optimized objective is $0.24+0.5(0.724719)\approx0.602359$, which agrees with $-\tau\log Z_\tau=0.5(1.204719)\approx0.602359$.

Temperature and the Partition Function

Writing $\beta=1/\tau$ makes clear that $\beta$ is an inverse temperature: the density is proportional to $\exp\{-\beta\widehat R_n(h)\}$. This is a convention, so a statement that “higher temperature concentrates the posterior” would reverse the direction. If the empirical risk is bounded, the response of the normalizer to $\beta$ can be derived directly.

Result: If $\widehat R_n$ is bounded and measurable, then $$ \frac{d}{d\beta}\log Z_\beta =-E_{Q_\beta}[\widehat R_n], \qquad \frac{d^2}{d\beta^2}\log Z_\beta =\operatorname{Var}_{Q_\beta}(\widehat R_n)\geq0, $$ where $Z_\beta=\int e^{-\beta\widehat R_n}\,d\Pi$ and $dQ_\beta/d\Pi=e^{-\beta\widehat R_n}/Z_\beta$.

Derivation. Boundedness of $\widehat R_n$ permits differentiation under the integral: on any bounded interval of $\beta$, the derivatives of $e^{-\beta\widehat R_n}$ through second order are bounded by finite constants, which are integrable under the probability measure $\Pi$. Thus $Z_\beta'=-\int\widehat R_n e^{-\beta\widehat R_n}\,d\Pi$, and division by $Z_\beta$ yields the first identity. Differentiating the resulting expectation, or applying the quotient rule to $Z_\beta'/Z_\beta$, gives $(\log Z_\beta)''=E_{Q_\beta}[\widehat R_n^2]-E_{Q_\beta}[\widehat R_n]^2$, the stated variance. $\square$

The first derivative says that increasing inverse temperature decreases the log normalizer at a rate given by posterior-averaged empirical risk. The second derivative shows that $\log Z_\beta$ is convex. These facts describe the temperature response, but do not imply that every increase in concentration improves population performance.

Worked Example: A Concentrated Posterior and Its KL Cost

Consider three predictors with uniform prior mass and $\tau=0.1$. Let their empirical risks be $0,d,d$, where $d=0.1\log(1.9786/0.0107)\approx0.522$. These are valid risks in $[0,1]$. The unnormalized Gibbs weights, after removing their common prior factor, are $1,e^{-d/0.1},e^{-d/0.1}$.

By the choice of $d$, $e^{-d/0.1}=0.0107/1.9786$. Normalizing gives posterior probabilities $0.9893,0.00535,0.00535$, since their sum is one. The KL divergence from the uniform prior is $$ 0.9893\log(3\cdot0.9893) +2(0.00535)\log(3\cdot0.00535) \approx1.0320. $$ The posterior strongly favors the zero-risk predictor, but this concentration has a substantial prior-divergence cost.

Reading the Gibbs Posterior in a PAC-Bayes Bound

To connect the optimization to generalization, suppose $Z_1,\ldots,Z_n$ are i.i.d. from a common distribution $P$, and the loss $\ell(h,Z)$ is jointly measurable and lies in $[0,1]$. Define $R(h)=E_{Z\sim P}[\ell(h,Z)]$, $\widehat R_n(h)=n^{-1}\sum_{i=1}^n\ell(h,Z_i)$, and the averaged risks $R(Q)=\int R(h)\,dQ(h)$ and $\widehat R_n(Q)=\int\widehat R_n(h)\,dQ(h)$. Let $\operatorname{kl}(p,q)=p\log(p/q)+(1-p)\log\{(1-p)/(1-q)\}$, with its standard boundary conventions.

The PAC-Bayes-kl inequality states that, for a fixed prior $\Pi$ independent of the sample, with probability at least $1-\delta$ over the i.i.d. sample, simultaneously for every posterior $Q$, $$ \operatorname{kl}\bigl(\widehat R_n(Q),R(Q)\bigr) \leq \frac{\mathrm{KL}(Q\|\Pi)+\log(2\sqrt n/\delta)}{n}. $$ The Gibbs posterior is allowed to depend on the sample: the simultaneous quantifier permits substituting $Q_\tau$ after observing the data. The i.i.d. assumption matters because $R(h)$ here is a single common-distribution population risk.

Worked Example: Substituting a Gibbs Posterior Into the Bound

Use the three-predictor Gibbs posterior above, whose KL divergence is approximately $1.0320$. Suppose $n=100$ i.i.d. observations and choose $\delta=0.05$. The PAC-Bayes-kl right side is $$ \frac{1.0320+\log(2\sqrt{100}/0.05)}{100} = \frac{1.0320+\log(400)}{100} \approx \frac{1.0320+5.9915}{100} \approx0.0702. $$ Thus, with probability at least $0.95$, the binary KL divergence between the posterior-averaged empirical and population risks is at most about $0.0702$.

Pinsker’s inequality gives $|p-q|\leq\sqrt{\operatorname{kl}(p,q)/2}$, so in this example the absolute difference between those averaged risks is at most approximately $\sqrt{0.0702/2}=0.1874$. As a separate arithmetic check, if a posterior at the same $n,\delta$ had $\mathrm{KL}(Q\|\Pi)=0.1659$, the right side would be $(0.1659+\log400)/100\approx0.0616$, and the corresponding Pinsker bound would be approximately $\sqrt{0.0616/2}=0.1755$.

The bound uses the posterior’s KL divergence as a complexity term; the Gibbs variational identity explains why that term is balanced against empirical risk in the posterior construction. Temperature does not disappear: it controls the trade-off that produced $Q_\tau$. Choosing a temperature appropriate to a particular bound is a distinct step, not a consequence of the normalization formula alone.

Common Pitfall

A frequent error is to write $Q_\tau(h)\propto\exp\{-\widehat R_n(h)/\tau\}$ and treat the expression as a probability distribution without specifying its reference measure or normalizer. On a finite space this can omit prior weights; on a general space it may not even define a density with respect to the intended measure. The correction is to state the density relative to $\Pi$ and divide by $Z_\tau$. Another error is to read $Z_\tau$ itself as a generalization guarantee: it is a normalizing constant and determines the minimum variational objective, whereas generalization follows from an appropriate PAC-Bayes bound with its sampling and loss assumptions.

Key takeaway: The Gibbs posterior is the unique balance of empirical fit and KL proximity to the prior: $Q_\tau$ minimizes $\widehat R_n(Q)+\tau\mathrm{KL}(Q\|\Pi)$. Its partition function normalizes the posterior and gives the minimum objective value; a PAC-Bayes bound then uses the posterior’s empirical risk and KL cost to control population risk.

Check Your Understanding

Use the definitions and results above to answer the following questions.

For a fixed empirical risk and prior, how does the Gibbs density change when $\tau$ decreases?
Why does the variational identity imply that $Q_\tau$ is the unique minimizer?
In the Gaussian example, which terms determine the posterior mean and variance after completing the square?
Why is the partition function not, by itself, a PAC-Bayes generalization guarantee?
Which sampling assumption supports the stated single-population-risk PAC-Bayes-kl inequality, and why is independence alone insufficient for that formulation?