Choosing the PAC-Bayes Temperature

Why Temperature Is a Statistical Choice

In an exponential PAC-Bayes bound, temperature is not merely a tuning convention. It controls how empirical risk and the cost of using a posterior jointly enter the guarantee. A small temperature and a large temperature can therefore produce different bounds for the same data and posterior. Choosing between them is part of interpreting the guarantee, not just a numerical adjustment.

The relevant optimization keeps the posterior fixed and asks which admissible temperature gives the smallest bound. This distinction matters: changing the posterior as well as the temperature is a different optimization problem. In particular, the Gibbs posterior constructed from empirical risk is a separate topic.

The Bound as a Function of Temperature

Let $Z_1,\ldots,Z_n$ be independent observations from a distribution $P$, and let $\mathcal H$ be a hypothesis space. Assume the loss satisfies $0\leq \ell(h,z)\leq 1$. Define the population risk and empirical risk of $h$ by $R(h)=E[\ell(h,Z)]$ and $\hat R(h)=n^{-1}\sum_{i=1}^n\ell(h,Z_i)$. For a distribution $\rho$ on hypotheses, write $R(\rho)=E_{h\sim\rho}[R(h)]$ and $\hat R(\rho)=E_{h\sim\rho}[\hat R(h)]$.

Let $\pi$ be a prior distribution on $\mathcal H$ that is independent of the observed sample, and let $\operatorname{KL}(\rho\Vert\pi)$ denote the Kullback–Leibler divergence. The Catoni variational PAC-Bayes bound gives the following guarantee: for fixed $\lambda>0$ and $0<\delta<1$, with probability at least $1-\delta$ over the sample, every posterior $\rho$ satisfies

R(\rho)\leq B_\lambda(\rho) = \frac{1-\exp\!\left(-\lambda\hat R(\rho)-c_\rho\right)} {1-\exp(-\lambda)}, \qquad c_\rho=\frac{\operatorname{KL}(\rho\Vert\pi)+\log(1/\delta)}{n}.

Here $c_\rho\geq 0$ is the normalized complexity, including the confidence cost. The guarantee is for a specified temperature $\lambda$; the event is simultaneous over posteriors $\rho$, not automatically over all temperatures. The temperature is positive, and the loss range is the unit interval under this parameterization.

Definition: For fixed data and posterior $\rho$, the temperature-optimized Catoni bound is the infimum of $B_\lambda(\rho)$ over admissible $\lambda>0$, provided the PAC-Bayes guarantee remains valid for the way $\lambda$ is chosen. Write $r=\hat R(\rho)$ and $c=c_\rho$; the numerical optimization concerns $B_\lambda(r,c)=[1-\exp(-\lambda r-c)]/[1-\exp(-\lambda)]$.

This formula shows the rescaling directly. The empirical risk appears multiplied by $\lambda$, while the complexity $c$ is added in the exponent and is not multiplied by $\lambda$ in this convention. Temperature also changes the denominator $1-\exp(-\lambda)$. It is therefore generally incorrect to think of increasing $\lambda$ as simply increasing or decreasing one isolated penalty.

How the Minimizing Temperature Is Characterized

The derivative of the bound yields a one-dimensional equation for an interior optimum. The theorem below gives both the stationarity condition and the conditions under which it specifies a unique finite minimum.

Theorem: Suppose $0<r<1$ and $c>0$. The function $B_\lambda(r,c)$ has a unique minimizer over $\lambda>0$. It is the unique positive solution of $$ \log\!\left(1+r\bigl(e^\lambda-1\bigr)\right)-r\lambda=c. $$

Proof. Put $N(\lambda)=1-e^{-c-\lambda r}$ and $D(\lambda)=1-e^{-\lambda}$. Both are positive for $\lambda>0$, and $B_\lambda=N(\lambda)/D(\lambda)$. Differentiating gives

B'_\lambda = \frac{r e^{-c-\lambda r}(1-e^{-\lambda}) -(1-e^{-c-\lambda r})e^{-\lambda}} {(1-e^{-\lambda})^2}.

Multiplying the numerator by the positive quantity $e^{c+\lambda r+\lambda}$ shows that its sign is the sign of $r(e^\lambda-1)-(e^{c+\lambda r}-1)$. Thus $B'_\lambda=0$ exactly when $e^{c+\lambda r}=1+r(e^\lambda-1)$, which is the stated equation. Define $g(\lambda)=\log(1+r(e^\lambda-1))-r\lambda$. We have $g(0)=0$, and

g'(\lambda) = \frac{r(1-r)(e^\lambda-1)} {1-r+r e^\lambda}>0 \qquad(\lambda>0).

Also $g(\lambda)$ tends to infinity as $\lambda$ tends to infinity: since $\log(1+r(e^\lambda-1))-\lambda$ tends to $\log r$, we have $g(\lambda)-(1-r)\lambda\to\log r$. Consequently $g(\lambda)=c$ has exactly one positive solution. The derivative sign found above is the sign of $1+r(e^\lambda-1)-e^{c+\lambda r}$, which is positive when $g(\lambda)>c$ and negative when $g(\lambda)<c$. Since $g$ increases from $0$ through $c$, $B'_\lambda<0$ before the solution and $B'_\lambda>0$ after it. The solution is therefore the unique minimizer. $\square$

Boundary Cases and Limiting Temperatures

The theorem’s assumptions are important. When $c=0$, the bound behaves differently near zero; when $r$ is at an endpoint, there need not be a finite minimizer. The limits make those cases explicit.

Result: For $0\leq r\leq1$ and $c\geq0$, $$ \lim_{\lambda\downarrow0}B_\lambda(r,c)= \begin{cases} +\infty,&c>0,\\ r,&c=0, \end{cases} \qquad \lim_{\lambda\to\infty}B_\lambda(r,c)= \begin{cases} 1,&r>0,\\ 1-e^{-c},&r=0. \end{cases} $$

Derivation. As $\lambda\downarrow0$, the denominator satisfies $1-e^{-\lambda}\sim\lambda$, while the numerator is $1-e^{-c}e^{-\lambda r}$. If $c>0$, the numerator tends to $1-e^{-c}>0$, so the ratio diverges. If $c=0$, both numerator and denominator vanish; using $1-e^{-\lambda r}\sim\lambda r$ gives limit $r$, including $r=0$. As $\lambda\to\infty$, the denominator tends to $1$. For $r>0$, the numerator tends to $1$; for $r=0$, it is constantly $1-e^{-c}$. These give the stated limits. $\square$

When $c>0$ and $r=0$, $B_\lambda=(1-e^{-c})/(1-e^{-\lambda})$ decreases to $1-e^{-c}$; the infimum is approached as $\lambda\to\infty$, not attained at finite temperature. When $r=1$ and $c>0$, $B_\lambda$ also decreases, with limit $1$. When $c=0$ and $0<r<1$, the derivative calculation above has $g(\lambda)>0$ for every $\lambda>0$, so the bound increases and its infimum $r$ is approached as $\lambda\downarrow0$. If $c=0$ and $r=0$, the bound is identically zero; if $c=0$ and $r=1$, it is identically one.

Worked Temperature Comparisons

Worked Example: Comparing Two Temperatures

Suppose the fixed posterior has empirical risk $r=0.12$ and normalized complexity $c=0.049957$. Compare $\lambda=0.5$ and $\lambda=2$.

At $\lambda=0.5$, the exponent is $-0.5(0.12)-0.049957=-0.109957$, and the denominator is $1-e^{-0.5}\approx0.39347$. Therefore

B_{0.5} = \frac{1-e^{-0.109957}}{0.39347} \approx \frac{0.10413}{0.39347} \approx 0.2646.

At $\lambda=2$, the exponent is $-2(0.12)-0.049957=-0.289957$, and the denominator is $1-e^{-2}\approx0.86466$. Hence

B_2 = \frac{1-e^{-0.289957}}{0.86466} \approx \frac{0.25170}{0.86466} \approx 0.2911.

The smaller of these two evaluated bounds is $0.2646$. This does not establish that $\lambda=0.5$ is the optimum: the theorem’s stationarity equation, or a numerical minimization over $\lambda$, is needed to locate the minimum.

Worked Example: How Greater Complexity Changes the Comparison

Keep $r=0.12$, but increase the normalized complexity to $c=0.060943$. At $\lambda=0.5$, the exponent is $-0.06-0.060943=-0.120943$, so

B_{0.5} = \frac{1-e^{-0.120943}}{1-e^{-0.5}} \approx \frac{0.11392}{0.39347} \approx 0.2895.

At $\lambda=2$, the exponent is $-0.24-0.060943=-0.300943$, giving

B_2 = \frac{1-e^{-0.300943}}{1-e^{-2}} \approx \frac{0.25991}{0.86466} \approx 0.3006.

Both bounds are larger than their counterparts in the preceding example, where $r$ and the temperatures were unchanged but $c$ was smaller. This is the expected direction: increasing the divergence or confidence cost increases the bound at any fixed positive temperature.

Worked Example: Constructing a Known Optimal Temperature

Take $r=0.2$ and choose $c=\log(1+0.2(e-1))-0.2$, which is approximately $0.09539$. Substituting $\lambda=1$ into the stationarity equation gives

\log(1+0.2(e^1-1))-0.2(1)=c.

Thus $\lambda=1$ is the unique minimizer by the theorem. The bound at that temperature is

B_1 = \frac{1-\exp\{-0.2-c\}}{1-e^{-1}} \approx \frac{1-e^{-0.29539}}{0.63212} \approx \frac{0.25576}{0.63212} \approx 0.4046.

This construction checks both the optimization condition and the resulting bound. In applications, the equation for the minimizer is usually solved numerically because it has no general closed-form solution.

Admissibility: When May Temperature Be Chosen From the Data?

For a fixed, pre-specified $\lambda$, the stated PAC-Bayes guarantee applies with probability at least $1-\delta$. The posterior may still be selected using the data because the guarantee holds simultaneously over posteriors. But that uniformity over $\rho$ does not itself authorize selecting $\lambda$ after inspecting the same data and then quoting the fixed-$\lambda$ guarantee unchanged.

A valid procedure must account for temperature selection. One option is to prove a bound uniform over the relevant temperatures. Another is to choose from a countable grid $\{\lambda_j\}$, allocate failure probabilities $\delta_j>0$ with $\sum_j\delta_j\leq\delta$, and apply the bound at each grid point with confidence level $\delta_j$. A union bound then gives simultaneous validity over that grid. The resulting complexity term uses $\log(1/\delta_j)$ at temperature $\lambda_j$, so the selection cost is not free. A temperature fixed independently of the sample avoids this particular selection issue.

A further practical check is that a valid risk bound cannot improve on the trivial upper bound $1$ for a loss in $[0,1]$. If the displayed Catoni expression exceeds $1$, report the smaller of that expression and $1$; the latter is valid from the loss assumption alone. This clipping does not alter the derivative analysis of the uncapped expression where it gives a bound below $1$.

Common Mistake

Wrong reasoning: “The posterior is allowed to depend on the data, so I can choose the best temperature on the same data and retain the fixed-temperature confidence statement.”
Correction: The stated theorem is uniform over posteriors for a fixed temperature. Temperature selection requires a guarantee uniform over the selected set, or a valid selection adjustment such as a union bound over a grid. Keep the risk, complexity, temperature, and confidence cost in the same formula when comparing candidates.

Key takeaway: For fixed empirical Gibbs risk $r$ and normalized complexity $c$, an interior optimum with $0<r<1$ and $c>0$ is the unique positive solution of $\log(1+r(e^\lambda-1))-r\lambda=c$. Boundary cases require checking limits, and the PAC-Bayes guarantee must remain valid for the temperature-selection procedure.

Check Your Understanding

Answer each question using the temperature parameterization in this tutorial.

In $B_\lambda(r,c)$, which term multiplies empirical risk, and where does normalized complexity appear?
State the equation satisfied by the unique minimizing temperature when $0<r<1$ and $c>0$.
For $c>0$, what happens to the bound as $\lambda\downarrow0$, and why?
When $r=0$ and $c>0$, is the infimum attained at a finite positive temperature?
Why does a guarantee uniform over posteriors not by itself justify choosing temperature after looking at the data?