From Candidate Losses to an Aggregate
With two regression candidates, aggregation can be calculated directly rather than treated as a black-box procedure. The calculation has three distinct parts: measure each candidate's empirical squared loss, convert those losses into normalized exponential weights, and average the candidates' predictions using those weights. The aggregate's squared risk must then be calculated from its own predictions; it is not generally the weighted average of the candidates' risks.
We use a positive temperature \(T\). A smaller \(T\) makes differences in empirical loss matter more, while a larger \(T\) makes the weights less concentrated. Throughout, empirical risk means an average over the \(n\) observed cases. This tutorial concerns that finite-sample calculation; a low empirical risk alone does not establish low risk on new observations.
The weights are nonnegative and sum to one, so the aggregate is a convex combination of the two predictions. Unless stated otherwise, the examples below use equal prior weights, \(\pi_1=\pi_2=1/2\). Their common factor cancels in normalization. Write \(q=w_2(T)\); then \(w_1(T)=1-q\). Keeping track of which weight is assigned to which candidate is essential: a smaller empirical risk must produce the larger exponential weight.
Worked Calculation Across Temperatures
Worked Example: Two Candidates on Two Observations
Suppose the observed responses are \(y_1=0,y_2=1\), and the candidates predict \(f_1(x_1)=f_1(x_2)=0\) and \(f_2(x_1)=f_2(x_2)=2\). Calculate the empirical risks, the weights, and the aggregate's empirical squared risk at several temperatures.
For candidate 1, the squared errors are \(0^2=0\) and \(1^2=1\), so \(R_n(f_1)=(0+1)/2=1/2\). For candidate 2, the squared errors are \((0-2)^2=4\) and \((1-2)^2=1\), so \(R_n(f_2)=(4+1)/2=5/2\). Candidate 1 is better by 2 units of empirical risk.
With equal prior weights, the common factor \(1/2\) cancels. The weight on candidate 2 is therefore $$ q(T)=\frac{\exp\{-(5/2)/T\}} {\exp\{-(1/2)/T\}+\exp\{-(5/2)/T\}} =\frac{1}{1+\exp(2/T)}. $$ At each observation, the aggregate predicts \(0(1-q)+2q=2q\). Its empirical risk is $$ R_n(f_T)=\frac{(0-2q)^2+(1-2q)^2}{2} =\frac{4q^2+(1-2q)^2}{2} =\frac12-2q+4q^2. $$
Substituting temperatures gives the following values. The last column is the risk of the aggregate, not the weighted average of the two candidate risks.
| Temperature \(T\) | Weight \(q\) on candidate 2 | Aggregate prediction \(2q\) | Aggregate empirical risk |
|---|---|---|---|
| 0.5 | \(1/(1+e^4)\approx 0.0180\) | \(\approx 0.0360\) | \(\approx 0.4653\) |
| 1 | \(1/(1+e^2)\approx 0.1192\) | \(\approx 0.2384\) | \(\approx 0.3184\) |
| 2 | \(1/(1+e)\approx 0.2689\) | \(\approx 0.5379\) | \(\approx 0.2514\) |
| \(T\to\infty\) | \(0.5\) | \(1\) | \(0.5\) |
For example, when \(T=1\), \(q\approx0.1192\), so \(R_n(f_T)=0.5-2(0.1192)+4(0.1192)^2\approx0.3184\). As \(T\) tends to zero, the weight on the worse candidate tends to zero, and the aggregate approaches candidate 1, whose risk is \(0.5\). As \(T\) tends to infinity, the equal prior weights dominate and the aggregate prediction is 1 at both observations, also giving risk \(0.5\).
In this example the aggregate can do better than either candidate. The risk expression is minimized over \(q\) at \(q=1/4\), since $$ \frac12-2q+4q^2=4\left(q-\frac14\right)^2+\frac14. $$ The exponential weights attain \(q=1/4\) when \(1+\exp(2/T)=4\), or \(T=2/\log 3\approx1.8205\). At that temperature the aggregate risk is \(1/4\), below both candidate risks of \(1/2\). This is a property of averaging predictions under squared loss, not a claim that every temperature improves on the best candidate.
Two Exact Identities
The first identity makes the effect of temperature transparent. It is useful to calculate the weight ratio before normalizing, because the ratio immediately gives both the direction and strength of the preference.
Derivation. By definition, both weights have the same positive normalizing denominator. Dividing their formulas cancels it: $$ \frac{w_1(T)}{w_2(T)} =\frac{\pi_1\exp\{-R_n(f_1)/T\}} {\pi_2\exp\{-R_n(f_2)/T\}} =\frac{\pi_1}{\pi_2} \exp\left\{\frac{R_n(f_2)-R_n(f_1)}{T}\right\}. $$ The exponential factor exceeds one if and only if \(R_n(f_2)>R_n(f_1)\). With equal priors, this makes the ratio exceed one, proving the stated comparison. As \(T\) increases, the loss difference is divided by a larger number, so the exponential factor moves closer to one. \(\square\)
The second identity separates the weighted average of candidate risks from the risk of the averaged prediction. It holds observation by observation, so it also holds after averaging across the data.
Proof. Fix an observation \(i\), and abbreviate \(a=f_1(x_i)\), \(b=f_2(x_i)\), and \(y=y_i\). Since \(f_w(x_i)=wa+(1-w)b\), expanding the right-hand side of the following identity gives $$ w(y-a)^2+(1-w)(y-b)^2-w(1-w)(a-b)^2 =\bigl(y-wa-(1-w)b\bigr)^2. $$ For detail, the coefficients of \(y^2\) on both sides are 1. The coefficient of \(y\) on the left is \(-2wa-2(1-w)b\), which matches the right. The remaining terms on the left simplify to $$ wa^2+(1-w)b^2-w(1-w)(a^2-2ab+b^2) =\bigl(wa+(1-w)b\bigr)^2. $$ This is the remaining term on the right. The identity therefore holds for each \(i\). Summing it over \(i=1,\ldots,n\) and dividing by \(n\) gives the theorem. The subtracted term is nonnegative because it is a nonnegative weight times an average of squares. \(\square\)
For exponential aggregation, set \(w=w_1(T)\) and \(1-w=w_2(T)\). The theorem explains why computing \((1-w)R_n(f_1)+wR_n(f_2)\) does not answer the question “What is the risk of the averaged prediction?” The latter also depends on how far apart the candidates' predictions are on the observed inputs. If the two candidates make identical predictions on every observed input, the correction term is zero; otherwise it is nonnegative and can be substantial.
Two Further Hand Checks
Worked Example: A Single Observation and the Risk Correction
There is one observation with response \(y=0\). Candidate 1 predicts 1 and candidate 2 predicts 3. Their empirical risks are \(1\) and \(9\), respectively. With equal prior weights and \(T=4\), the weight on candidate 2 is $$ q=\frac{e^{-9/4}}{e^{-1/4}+e^{-9/4}} =\frac{1}{1+e^2}\approx0.1192. $$
The aggregate predicts \(1(1-q)+3q=1+2q\approx1.2384\), so its squared risk is \((1+2q)^2\approx1.5336\). By contrast, the weighted average of the candidate risks is \((1-q)1+q9=1+8q\approx1.9536\). The difference is $$ (1+8q)-(1+2q)^2=4q(1-q)\approx0.4200. $$ This equals \(q(1-q)(3-1)^2\), precisely the correction term in the theorem. The aggregate risk is lower than the weighted candidate risk, but it is still greater than candidate 1's risk of 1 at this temperature.
Worked Example: Equal Candidate Risks at Every Temperature
Suppose \(n=2\), the responses are \(y_1=0,y_2=1\), and the predictions are \(f_1(x_1)=f_1(x_2)=0\) and \(f_2(x_1)=0,f_2(x_2)=1\). Candidate 1 has squared errors \(0,1\), while candidate 2 has squared errors \(0,0\). To instead make the losses tie, consider candidate 2 predictions \(f_2(x_1)=1,f_2(x_2)=1\): its squared errors are \(1,0\). Thus both empirical risks equal \(1/2\).
With equal prior weights, equal risks give equal exponential factors, so \(w_1(T)=w_2(T)=1/2\) for every \(T>0\). The aggregate predictions are \(1/2\) at each observation. Its empirical risk is $$ R_n(f_T)=\frac{(0-1/2)^2+(1-1/2)^2}{2} =\frac{1/4+1/4}{2}=\frac14. $$ Here the candidates tie in empirical risk, but their distinct predictions allow the average to achieve smaller empirical risk. In particular, temperature cannot distinguish candidates with equal risks under equal priors; it does not change their relative weights.
Common Calculation Pitfalls
- Reversing the loss difference. A formula such as \(\exp\{(R_n(f_1)-R_n(f_2))/T\}\) for \(w_1/w_2\) has the wrong sign. The correct ratio contains \(R_n(f_2)-R_n(f_1)\), so a lower risk gives a larger weight.
- Forgetting normalization. The unnormalized quantities \(\pi_j\exp\{-R_n(f_j)/T\}\) are not themselves probabilities or weights summing to one. Divide each by their sum before averaging predictions.
- Confusing average risk with aggregate risk. The weighted average of candidate risks omits the nonnegative squared-distance correction. Compute the aggregate predictions first, or use the exact decomposition.
- Interpreting a large temperature as stronger selection. The exponent divides the risk by \(T\). A larger \(T\) flattens the weights toward the prior proportions; a smaller \(T\) emphasizes the empirical-risk difference.
- Changing between total and average loss without adjusting temperature. If losses are summed rather than averaged, their values are multiplied by \(n\). To preserve the same weights, the temperature must also be multiplied by \(n\).
A reliable hand calculation therefore records the two empirical risks, writes the unnormalized weights with the same temperature and loss scale, normalizes them, and then calculates the aggregate prediction and its risk. The decomposition provides a final check: the risk of the aggregate cannot exceed the weighted average of candidate risks under squared loss, though it need not be below the risk of the better candidate.
Check Your Understanding
Work through these questions by writing the losses and weights explicitly before simplifying.
- If \(R_n(f_1)=2\), \(R_n(f_2)=5\), and the prior weights are equal, which candidate has the larger exponential weight? Write the ratio \(w_1(T)/w_2(T)\).
- For the data in the first worked example, what are the aggregate's predictions when \(T=2\), and how does their empirical risk compare with the risk of candidate 1?
- State the squared-loss decomposition for an aggregate with weight \(w\) on candidate 1. What makes its correction term nonnegative?
- If both candidate risks are equal and their prior weights are equal, how do their normalized weights depend on temperature?
- Suppose weights are computed from total squared loss rather than average squared risk. By what factor must the temperature change to leave the normalized weights unchanged?