Computing Gibbs Weights for a Finite Regression Dictionary

From Candidate Losses to Gibbs Weights

A finite regression dictionary consists of candidate predictors \(f_1,\ldots,f_M\). After observing a sample, each candidate has an empirical loss, and a Gibbs rule assigns it weight according to both that loss and a prior preference. The computation is simple once the convention is fixed, but mixing average loss with cumulative loss changes the weights and the oracle penalty.

Let \((x_i,y_i)\), \(i=1,\ldots,n\), be a fixed sample, and let \(\ell(y,a)\) be the loss for outcome \(y\) and prediction \(a\). Write \(L_j=\sum_{i=1}^n\ell(y_i,f_j(x_i))\) for candidate \(j\)'s cumulative loss and \(R_j=L_j/n\) for its average empirical loss. Let \(\pi=(\pi_1,\ldots,\pi_M)\) be a prior distribution on the candidates, with \(\pi_j>0\) and \(\sum_j\pi_j=1\). For temperature \(\eta>0\), the Gibbs weights are

q_j=\frac{\pi_j e^{-\eta L_j}}{\sum_{k=1}^M \pi_k e^{-\eta L_k}} =\frac{\pi_j e^{-\eta n R_j}}{\sum_{k=1}^M \pi_k e^{-\eta n R_k}}.

The denominator normalizes the weights to sum to one. A candidate with lower loss receives more weight, while a larger prior mass also increases its weight. The aggregate prediction at a new input \(x\) is the weighted average

\hat f(x)=\sum_{j=1}^M q_j f_j(x).

This is a fixed-sample aggregate: the weights use the losses from all \(n\) observations. It is distinct from a sequential procedure that updates weights as observations arrive.

Definition: The Gibbs distribution on a finite dictionary is the probability vector \(q\) with coordinates proportional to prior mass times an exponential loss factor, \(q_j\propto \pi_j e^{-\eta L_j}\). The quantity \(\eta\) is the inverse-temperature parameter: increasing \(\eta\) makes the distribution more sensitive to loss differences.

Worked Calculations

Worked Example: A Nonuniform Prior and an Aggregate Prediction

There are three constant regression candidates, evaluated on \(n=3\) observations with response \(y_i=0\) each time. Their predictions are \(f_1(x)=0\), \(f_2(x)=1/\sqrt{3}\), and \(f_3(x)=\sqrt{2/3}\). Use squared loss, prior \(\pi=(1/2,3/10,1/5)\), and \(\eta=\log 2\).

For a constant prediction \(c\), each squared loss is \(c^2\), so the cumulative losses are \(L_1=0\), \(L_2=3(1/3)=1\), and \(L_3=3(2/3)=2\). The unnormalized Gibbs weights are therefore

\left(\frac12,\frac{3}{10}\,2^{-1},\frac15\,2^{-2}\right) =\left(\frac12,\frac{3}{20},\frac1{20}\right).

Their sum is \(7/10\), giving \(q=(5/7,3/14,1/14)\). Since these candidates are constant, the aggregate is constant too:

\hat f(x)=\frac{5}{7}(0)+\frac{3}{14\sqrt{3}}+\frac{1}{14}\sqrt{\frac23} =\frac{\sqrt{3}+\sqrt{2/3}}{14} \approx 0.1820.

Its empirical squared loss is \(\hat f(x)^2\approx 0.0331\) on each observation, hence also approximately \(0.0331\) when averaged over the three identical observations. The posterior-weighted average candidate risk is \((3/14)(1/3)+(1/14)(2/3)=5/42\approx0.1190\); the aggregate's loss is smaller, as convexity permits.

Worked Example: Prior Odds and a Query Prediction

Two candidates have cumulative losses \(L_1=1\) and \(L_2=2\). Their prior probabilities are \(\pi_1=1/4\) and \(\pi_2=3/4\), and take \(\eta=\log 2\). At a query point \(x_*\), the predictions are \(f_1(x_*)=0.2\) and \(f_2(x_*)=0.8\).

The unnormalized weights are \((1/4)2^{-1}=1/8\) and \((3/4)2^{-2}=3/16\). The normalizing constant is \(1/8+3/16=5/16\), so \(q_1=2/5\) and \(q_2=3/5\). The query prediction is

\hat f(x_*)=\frac25(0.2)+\frac35(0.8)=0.08+0.48=0.56.

The posterior odds also provide a quick check: \(q_1/q_2=(\pi_1/\pi_2)e^{-\eta(L_1-L_2)} =(1/3)e^{\log 2}=2/3\), which agrees with \((2/5)/(3/5)\). This odds calculation is often safer than normalizing several rounded decimal values.

Worked Example: Changing Temperature and Loss Scaling

Take prior \((1/2,3/10,1/5)\) and cumulative losses \((0,2,4)\). At \(\eta=(\log 2)/2\), the exponential factors are \((1,1/2,1/4)\). The unnormalized weights are \((1/2,3/20,1/20)\), so the normalized weights are \((5/7,3/14,1/14)\).

If instead \(\eta=\log 2\), the factors become \((1,1/4,1/16)\). The unnormalized weights are then \((1/2,3/40,1/80)\), whose sum is \(47/80\); the normalized weights are \((40/47,6/47,1/47)\). The larger temperature concentrates more weight on the zero-loss candidate.

The same calculation can be written using average losses \(R_j=L_j/n\), but the exponent must then be \(-\eta nR_j\). Using \(-\eta R_j\) instead would silently remove the factor \(n\) and produce different weights. For example, with \(n=2\), \(L=(0,2,4)\), and \(\eta=(\log 2)/2\), the correct factors are \((1,1/2,1/4)\), not \((1,2^{-1/2},2^{-1})\).

Why the Prior Determines the Oracle Penalty

For these weights, the natural penalty for comparing with candidate \(j\) is proportional to \(\log(1/\pi_j)\). A candidate given little prior mass pays a larger price for serving as the comparator. The next result gives the exact finite-sample calculation behind this penalty.

Theorem: Let \(q_j=\pi_j e^{-\eta nR_j}/Z\), where \(Z=\sum_k\pi_k e^{-\eta nR_k}\), and let \(q\) and \(\pi\) be probability distributions with all \(\pi_j>0\). Then \[ \sum_jq_jR_j+\frac{1}{\eta n}\sum_jq_j\log\frac{q_j}{\pi_j} =-\frac{1}{\eta n}\log Z \le R_k+\frac{\log(1/\pi_k)}{\eta n} \] for every candidate \(k\).

Proof. From the definition of \(q_j\), \(\log(q_j/\pi_j)=-\eta nR_j-\log Z\). Multiplying by \(q_j\) and summing gives \(\sum_jq_j\log(q_j/\pi_j)=-\eta n\sum_jq_jR_j-\log Z\), because \(\sum_jq_j=1\). Rearranging proves the equality. Also \(Z\ge \pi_k e^{-\eta nR_k}\), so \(-\log Z\le \eta nR_k+\log(1/\pi_k)\). Dividing by \(\eta n>0\) proves the inequality. \(\square\)

The sum \(\sum_jq_j\log(q_j/\pi_j)\) is the Kullback–Leibler divergence \(\operatorname{KL}(q\|\pi)\), which is nonnegative. Dropping it from the left side therefore yields \(\sum_jq_jR_j\le R_k+\log(1/\pi_k)/(\eta n)\). This controls the posterior-weighted empirical risk. To transfer the control to the prediction averaged with those weights, a condition on the loss is needed.

Definition: A loss is \(\eta\)-exp-concave in the prediction on a convex set \(\mathcal A\) if, for every fixed outcome \(y\), the function \(a\mapsto e^{-\eta\ell(y,a)}\) is concave on \(\mathcal A\).

Corollary: Suppose each candidate prediction and their aggregate lie in a convex set \(\mathcal A\), and the loss is \(\eta\)-exp-concave there. For the fixed Gibbs weights above, \[ \frac1n\sum_{i=1}^n\ell\!\left(y_i,\sum_jq_jf_j(x_i)\right) \le \min_{1\le k\le M}\left\{R_k+\frac{\log(1/\pi_k)}{\eta n}\right\}. \]

Proof. For each observation \(i\), exp-concavity and \(\sum_jq_j=1\) give \[ e^{-\eta\ell(y_i,\sum_jq_jf_j(x_i))} \ge \sum_jq_je^{-\eta\ell(y_i,f_j(x_i))}. \] Taking negative logarithms and dividing by \(\eta\) shows that the aggregate loss is at most \(-\eta^{-1}\log\sum_jq_je^{-\eta\ell(y_i,f_j(x_i))}\). By Jensen's inequality for the convex exponential function, this is at most \(\sum_jq_j\ell(y_i,f_j(x_i))\). Sum over \(i\), divide by \(n\), and apply the theorem's bound for each \(k\). Taking the minimum over \(k\) proves the result. \(\square\)

When Squared Loss Meets the Condition

For squared loss \(\ell(y,a)=(y-a)^2\), suppose outcomes and predictions lie in an interval \([A,B]\) of width \(D=B-A>0\). For fixed \(y\), put \(g(a)=e^{-\eta(y-a)^2}\). Direct differentiation gives

g''(a)=e^{-\eta(y-a)^2}\left(4\eta^2(a-y)^2-2\eta\right).

Since \((a-y)^2\le D^2\), this second derivative is nonpositive whenever \(0<\eta\le 1/(2D^2)\). Thus squared loss is \(\eta\)-exp-concave on that range, and the corollary gives the aggregate's empirical oracle bound with penalty \(\log(1/\pi_k)/(\eta n)\). The temperature used to compute weights must satisfy the condition for this particular guarantee; the formula for Gibbs weights itself is defined for any \(\eta>0\).

Common Mistakes

Confusing cumulative and average loss. With \(R_j=L_j/n\), the exponent is \(-\eta nR_j\), and the average-risk penalty is \(\log(1/\pi_j)/(\eta n)\). Omitting \(n\) changes both the weights and the bound.
Using a uniform-prior penalty with a nonuniform prior. The comparator-specific term is \(\log(1/\pi_j)\), not automatically \(\log M\). It equals \(\log M\) only for a uniform prior.
Assuming every temperature gives the squared-loss guarantee. The weights can always be computed, but the exp-concavity argument requires an admissible temperature and a prediction range on which its condition holds.
Treating the static aggregate as a sequential predictor. Here one posterior is computed from the full sample and then used in the aggregate. Sequential updates are a different procedure.

Key takeaway: Write down whether the exponent uses cumulative or average loss before calculating. Normalize \(\pi_j e^{-\eta L_j}\), average candidate predictions with the resulting weights, and use the prior-specific penalty \(\log(1/\pi_j)/(\eta n)\) only with the conditions that justify the desired risk bound.

Check Your Understanding

Use the definitions and calculations above to check each step of your reasoning.

If all candidate losses are equal, what are the Gibbs weights, regardless of temperature?
For two candidates, derive \(q_1/q_2\) in terms of their prior probabilities, cumulative losses, and \(\eta\).
Why does a candidate with smaller prior probability incur a larger oracle penalty?
If the loss is squared loss and predictions and outcomes lie in \([0,2]\), what upper bound on \(\eta\) ensures the exp-concavity condition used here?
What changes in the Gibbs exponent and oracle penalty when the empirical loss is written as an average rather than a cumulative sum?