Clipping and Tail Conditions for Squared-Loss Aggregation

Why Unbounded Responses Change the Guarantee

For squared loss, the response is part of the range of the loss. If $Y$ can be arbitrarily large, then even a predictor $p$ confined to a fixed interval can incur arbitrarily large loss $(Y-p)^2$. Consequently, a bounded-loss aggregation guarantee cannot be applied just by assuming the predictions are bounded. The problem also affects exp-concavity: the curvature of the exponential of negative squared loss depends on the distance between the response and prediction.

A useful remedy is to replace the response by a clipped version when constructing the aggregation weights. If the predictions are bounded too, the clipped loss has a finite range and a uniform exp-concavity constant. The price of this step is a clipping-bias term when the final performance is measured using the original, unclipped response. A tail or moment condition makes that term explicit.

Definition: For a threshold $B>0$, define $T_B(y)=\max\{-B,\min(y,B)\}$. The clipped response is $Y^B=T_B(Y)$. For any predictor value $p$, the clipped squared loss is $\ell_B(p;Y)=(Y^B-p)^2$, while the original squared loss is $\ell(p;Y)=(Y-p)^2$.

Throughout, suppose $(X_i,Y_i)$, $i=1,\ldots,n$, are independent and identically distributed. There are $M$ measurable candidate regression functions $g_1,\ldots,g_M$, each satisfying $|g_j(x)|\le B$ for all relevant $x$. Let $\pi_1,\ldots,\pi_M$ be positive prior weights summing to one. The predictor at each step is a mixture of the candidates, so it too lies in $[-B,B]$. The candidates may be fixed in advance; the weights, unlike the candidates, are updated using the observed data.

Worked Example: Why a Fixed Temperature Fails Without Clipping

For a fixed response $y$, consider $f(p)=\exp\{-\eta(y-p)^2\}$. Its second derivative is $f''(p)=\{4\eta^2(y-p)^2-2\eta\}\exp\{-\eta(y-p)^2\}$. For $f$ to be concave at $p$, it is necessary that $2\eta(y-p)^2\le 1$.

Take $\eta=1/8$, $y=10$, and $p=0$. Then $4\eta^2(y-p)^2-2\eta=4(1/64)(100)-1/4=6$, which is positive. Thus $f''(0)>0$, so this exponential loss is not concave there. With an unbounded response, $|y-p|$ can be arbitrarily large, and no fixed positive $\eta$ can make this curvature condition hold for every response and every bounded prediction.

For example, if $Y\sim N(0,1)$ and $p=0$, the loss $Y^2$ is unbounded: for every finite $t$, $P(Y^2>t)>0$. A bounded-loss guarantee therefore does not apply to this original loss merely because the predictor is bounded.

Sequential Aggregation with a Clipped Response

The sequential exponential-weights mixture updates its weights after each clipped observation. At step $i$, let $w_{i,j}$ be the normalized weight on candidate $j$, and predict $p_i(x)=\sum_{j=1}^M w_{i,j}g_j(x)$. Initialize with $w_{1,j}=\pi_j$, and update after observing $(X_i,Y_i)$ by multiplying each weight by $\exp\{-\eta(Y_i^B-g_j(X_i))^2\}$, then normalizing. In particular, the prediction at step $i$ uses only observations before step $i$.

Theorem: Suppose $|g_j(x)|\le B$, $\pi_j>0$, and $0<\eta\le 1/(8B^2)$. For any realized sequence of responses and covariates, the sequential mixture satisfies $$ \sum_{i=1}^n (Y_i^B-p_i(X_i))^2 \le \min_{1\le j\le M} \left\{ \sum_{i=1}^n (Y_i^B-g_j(X_i))^2 +\frac{\log(1/\pi_j)}{\eta} \right\}. $$

To prove the result, first note that $|Y_i^B-g_j(X_i)|\le 2B$. For fixed $y\in[-B,B]$, the second derivative of $p\mapsto \exp\{-\eta(y-p)^2\}$ is $$ \{4\eta^2(y-p)^2-2\eta\}\exp\{-\eta(y-p)^2\}. $$ When $|p|\le B$ and $\eta\le1/(8B^2)$, we have $2\eta(y-p)^2\le 8\eta B^2\le1$. The second derivative is therefore nonpositive, so this function is concave on $[-B,B]$. Jensen's inequality then gives $$ \exp\{-\eta(Y_i^B-p_i(X_i))^2\} \ge \sum_{j=1}^M w_{i,j}\exp\{-\eta(Y_i^B-g_j(X_i))^2\}. $$

Let $S_i=\sum_{j=1}^M\pi_j\exp\{-\eta\sum_{k=1}^i(Y_k^B-g_j(X_k))^2\}$, with $S_0=1$. The right side of the preceding inequality is $S_i/S_{i-1}$, by the weight update. Taking negative logarithms and summing over $i$ yields $$ \eta\sum_{i=1}^n(Y_i^B-p_i(X_i))^2 \le -\log S_n. $$ For each $j$, $S_n\ge\pi_j\exp\{-\eta\sum_{i=1}^n(Y_i^B-g_j(X_i))^2\}$. Hence $$ -\log S_n \le \eta\sum_{i=1}^n(Y_i^B-g_j(X_i))^2+\log(1/\pi_j). $$ This holds for every $j$; dividing by $\eta$ and minimizing the paired right-hand side proves the stated inequality. $\square$

From Clipped Risk to Original Risk

For a fixed predictor $g$ with values in $[-B,B]$, write $R(g)=E[(Y-g(X))^2]$ for its original risk and $R_B(g)=E[(Y^B-g(X))^2]$ for its clipped-response risk. Clipping the response cannot increase squared loss when the prediction is in the clipping interval. For instance, if $Y>B$, then $Y\ge B\ge g(X)$, so $(Y-g(X))^2\ge(B-g(X))^2$; the case $Y<-B$ follows in the same way. Thus $R_B(g)\le R(g)$.

The increase from clipped to original loss is zero when $|Y|\le B$. On the tail, it is nonnegative and at most $(|Y|+B)^2$, since the clipped loss is nonnegative and $|Y-g(X)|\le |Y|+B$. This gives a uniform bound that applies to every mixture prediction in the interval.

Definition: The response clipping-bias bound is $$ \delta_B=E\left[(|Y|+B)^2\mathbf{1}\{|Y|>B\}\right]. $$ If $E|Y|^q<\infty$ for some $q>2$, then $$ \delta_B\le \frac{4E|Y|^q}{B^{q-2}}. $$

Indeed, on $\{|Y|>B\}$, $(|Y|+B)^2\le4Y^2$, and $Y^2\le |Y|^q/B^{q-2}$. Taking expectations proves the moment bound. More generally, a finite second moment is enough to make $\delta_B$ finite, and $\delta_B\to0$ as $B\to\infty$: the integrand is bounded by $4Y^2\mathbf{1}\{|Y|>B\}$, whose expectation tends to zero by dominated convergence.

Theorem: Under the iid setup above, suppose $E[Y^2]<\infty$ and $0<\eta\le1/(8B^2)$. Let $p_i$ be the sequential mixture constructed from clipped responses. Its original-risk performance satisfies $$ E\left[\frac1n\sum_{i=1}^n (Y_i-p_i(X_i))^2\right] \le \min_{1\le j\le M} \left\{ R(g_j)+\frac{\log(1/\pi_j)}{\eta n} \right\} +\delta_B. $$

For the proof, apply the pathwise aggregation inequality and take expectations. Because the current observation is independent of the past that determines $p_i$, the expected clipped loss of $p_i$ at step $i$ is its clipped risk, conditional on that past. The expected clipped loss of candidate $g_j$ is $R_B(g_j)$. Thus the expected average clipped loss of the mixture is at most $$ \min_j\left\{R_B(g_j)+\frac{\log(1/\pi_j)}{\eta n}\right\}. $$ For each realized past and current prediction, the difference between original and clipped loss is at most $(|Y_i|+B)^2\mathbf{1}\{|Y_i|>B\}$. Its expectation is $\delta_B$. Therefore the expected average original loss is at most the expected average clipped loss plus $\delta_B$. Finally, $R_B(g_j)\le R(g_j)$ for every candidate. These inequalities give the stated bound. $\square$

The risk and complexity terms remain paired for the same candidate inside the minimum. In general, one cannot replace this minimum by the minimum risk plus the separately minimized complexity penalty: the candidate with the smallest risk need not have the smallest prior penalty.

Worked Example: Exact Clipping Bias Under a Two-Point Tail

Suppose $P(Y=0)=0.9$, $P(Y=4)=0.05$, and $P(Y=-4)=0.05$. Use $B=2$, and take any constant prediction $p\in[-2,2]$. At $Y=4$, the original-minus-clipped loss is $(4-p)^2-(2-p)^2=12-4p$. At $Y=-4$, it is $(-4-p)^2-(-2-p)^2=12+4p$.

The expected difference is therefore $0.05(12-4p)+0.05(12+4p)=1.2$, independent of $p$. The exact clipping bias for every such constant predictor is $1.2$. The general bound $\delta_2=E[(|Y|+2)^2\mathbf{1}\{|Y|>2\}]$ is $0.1(6^2)=3.6$, so it is valid but conservative. Here $E|Y|^4=0.1(4^4)=25.6$; the moment bound with $q=4$ gives $\delta_2\le4(25.6)/2^2=25.6$, looser still. The example shows why a directly evaluated tail bound can be more informative than a generic moment inequality.

Worked Example: Keeping Comparator Risk and Complexity Paired

Let $Y=0$ deterministically, $B=1$, $n=1$, and $\eta=1/8$. Use two constant candidates $g_1=0$ and $g_2=1$, with priors $\pi_1=0.01$ and $\pi_2=0.99$. There is no clipping bias. The first prediction is $p_1=0.01(0)+0.99(1)=0.99$, so its original loss is $0.99^2=0.9801$.

The two candidate risks are $R(g_1)=0$ and $R(g_2)=1$. The paired oracle bound has right-hand side $$ \min\left\{0+8\log(100),\ 1+8\log(1/0.99)\right\} = \min\{36.8414\ldots,\ 1.0804\ldots\} = 1.0804\ldots. $$ The actual loss $0.9801$ is below this bound. Separating the best risk from the smallest complexity penalty would instead give $0+8\log(1/0.99)\approx0.0804$, which is false for this example. The risk and penalty must be evaluated for the same comparator before minimizing.

Choosing the Threshold and Avoiding Misuse

Clipping creates a trade-off. A larger $B$ reduces the tail contribution $\delta_B$, but it also reduces the largest allowed temperature, since $\eta\le1/(8B^2)$. A smaller temperature increases the complexity term $\log(1/\pi_j)/(\eta n)$. The moment bound $\delta_B\le4E|Y|^q/B^{q-2}$ makes the tail side of this trade-off quantitative when a $q$-th moment is available; it does not by itself select an optimal threshold.

The theorem compares the sequential mixture's average performance across steps with the best single candidate, accounting for that candidate's prior weight. Its conclusion is an expected-risk statement under iid sampling, not a claim that every realized data set has small original loss. Also, the candidates must be bounded by $B$. If the original candidate functions are unbounded, replacing them by clipped functions changes the comparator and requires an additional approximation term; that change is not covered by the response clipping bias alone.

Wrong: bounded predictions imply bounded squared loss. Correction: the response must also be bounded, or the loss must be controlled by another argument.
Wrong: clipping removes all cost of heavy tails. Correction: the original-risk guarantee pays $\delta_B$, which depends on the response tail.
Wrong: minimize comparator risk and prior penalty separately. Correction: minimize their sum for the same candidate.
Wrong: apply the theorem to unbounded candidates without adjustment. Correction: ensure the candidates lie in $[-B,B]$, or account separately for prediction clipping.

Key takeaway: Clipping gives squared-loss aggregation a finite curvature scale when both the response used for updating and the candidates are bounded. Returning to original-response risk costs a tail term, and the oracle comparison keeps each candidate's risk and prior complexity together.

Check Your Understanding

Use the definitions and bounds above to answer these questions.

Why does boundedness of the predictors alone fail to make squared loss bounded when the response has unbounded support?
For $|Y^B|\le B$ and $|p|\le B$, show how the condition $\eta\le1/(8B^2)$ ensures concavity of $p\mapsto\exp\{-\eta(Y^B-p)^2\}$.
Under $E|Y|^q<\infty$, where $q>2$, how does the stated upper bound on $\delta_B$ change as $B$ increases?
Why must the risk and prior penalty remain paired for a single candidate inside the oracle minimum?
What extra issue arises if candidate functions are not bounded by the clipping threshold?

Clipping and Tail Conditions for Squared-Loss Aggregation

What You'll Learn