Tail Assumptions for Neural Regression Generalization

Why Squared Loss Needs a Tail Assumption

A squared-loss generalization bound compares empirical risk with population risk. For a predictor \(g\), define \(R(g)=E[(Y-g(X))^2]\) and, for independent identically distributed observations \((X_i,Y_i)\), define \(\widehat R_n(g)=n^{-1}\sum_{i=1}^n(Y_i-g(X_i))^2\). The difficulty is that a single large response can contribute an enormous squared loss. Consequently, a concentration inequality that works for bounded losses cannot be applied merely because the sample size is large or the predictor is a neural network.

The relevant assumptions concern the response and the resulting loss, not the network label. Bounded responses make the loss bounded when predictions are bounded. Truncation can create a bounded-loss problem while leaving an explicit tail error. A sub-Gaussian noise assumption supplies a tail bound that quantifies that error. Without some such control, even a response with finite variance can produce a sample that systematically misses rare, consequential observations.

Bounded Responses Give a Direct Bound

Let \(\mathcal G\) be a finite set of candidate predictors. Suppose \(|Y|\le M\) almost surely and \(|g(X)|\le B\) almost surely for every \(g\in\mathcal G\), where \(M,B\ge0\). Then every squared loss lies between \(0\) and \((M+B)^2\). Hoeffding's inequality therefore applies to each fixed predictor, and a union bound makes the result simultaneous over the finite set.

Proof. Fix \(g\) and write \(Z_i=(Y_i-g(X_i))^2\). The \(Z_i\) are independent, identically distributed, and lie in an interval of length \((M+B)^2\). If \(M+B>0\), Hoeffding's inequality gives, for \(t>0\),

The probability that this deviation exceeds \(t\) for at least one \(g\in\mathcal G\) is at most \(2|\mathcal G|\exp(-2nt^2/(M+B)^4)\). Set \(t=(M+B)^2\sqrt{\log(2|\mathcal G|/\delta)/(2n)}\). Substitution makes that probability at most \(\delta\), proving the claim. The zero case was handled separately. \(\square\)

Truncation Separates Concentration from Tail Error

If the response is not bounded, truncate it at a level \(\tau>0\): \(Y_\tau=\max(-\tau,\min(Y,\tau))\). Assume predictions satisfy \(|g(X)|\le B\), and choose \(\tau\ge B\). The truncated squared loss is bounded by \((\tau+B)^2\), so it concentrates. The difference between the original and truncated population risks is a tail term. This distinction matters: truncation does not make the original risk equal to the truncated risk; it makes the discrepancy measurable and, under suitable tails, small.

Proof. Let \(A\) be the event that \(|Y_i|\le\tau\) for all \(i\). The union bound gives \(P(A^c)\le nP(|Y|>\tau)\). On \(A\), every observed original response equals its truncated value, so \(\widehat R_n(g)=\widehat R_{\tau,n}(g)\) simultaneously for all \(g\). Because \(\tau\ge B\), clipping a response outside \([-\tau,\tau]\) moves it toward every prediction in \([-B,B]\). Thus \((Y-g(X))^2\ge(Y_\tau-g(X))^2\), and

The inequality follows because on the tail event the original squared loss is at most \((|Y|+B)^2\), while off that event the two losses agree. Apply the bounded-response theorem to the truncated losses, which lie in \([0,(\tau+B)^2]\), with failure probability \(\delta\). On its success event and on \(A\), the triangle inequality gives

Taking the supremum and combining the two event probabilities proves the stated bound. \(\square\)

Sub-Gaussian Noise Makes the Tail Term Explicit

A common regression model is \(Y=f^\star(X)+\varepsilon\), where \(|f^\star(X)|\le B\) almost surely and \(\varepsilon\) is conditionally sub-Gaussian given \(X\), with scale \(\sigma>0\). For \(u\ge0\), the event \(|Y|>B+u\) implies \(|\varepsilon|>u\). Hence \(P(|Y|>B+u)\le2\exp(-u^2/(2\sigma^2))\). This tail bound can be inserted into the truncation theorem. For example, if \(\tau>B\), put \(s=\tau-B>0\). Tail integration yields

To check the calculation, write \(W=|Y|\) and \(h(w)=(w+B)^2\). For \(\tau\ge0\), the tail-integration identity is \(E[h(W)\mathbf 1\{W>\tau\}]=h(\tau)P(W>\tau)+\int_\tau^\infty h'(w)P(W>w)\,dw\). Here \(h'(w)=2(w+B)\); substitute the response-tail bound and set \(u=w-B\). The integral is at most \(4\int_s^\infty(u+2B)e^{-u^2/(2\sigma^2)}du\). The first integral evaluates to \(\sigma^2e^{-s^2/(2\sigma^2)}\), and the second is bounded above by \((\sigma^2/s)e^{-s^2/(2\sigma^2)}\), giving the displayed expression. The other term in the truncation theorem is the bounded-loss concentration term at level \(\tau\). Increasing \(\tau\) reduces the tail term but increases that concentration term, so the threshold is a tradeoff rather than a free improvement.

Uncontrolled Tails Can Defeat Empirical Risk

Finite variance by itself does not guarantee a useful distribution-free concentration bound for squared loss. Consider a fixed predictor \(g=0\) and a response independent of \(X\), with \(Y=M\) with probability \(1/M^2\), and \(Y=0\) otherwise, for any \(M\ge1\). Then \(E[Y^2]=M^2/M^2=1\). Yet the empirical squared risk is zero whenever the sample contains no nonzero response, an event with probability \((1-1/M^2)^n\).

What the Assumptions Do—and Do Not—Buy

These results concern the tail behavior of squared loss and the search over predictors. Bounded responses and bounded predictions yield a direct finite-class bound. Truncation replaces an unbounded loss by a bounded one, but adds a population tail term and a probability that the observed sample contains a clipped response. Conditional sub-Gaussian noise makes those quantities controllable when the regression signal is bounded. None of these steps says that a particular neural network class has small complexity; controlling the class remains a separate part of a generalization argument.

• Common mistake: “The response has finite variance, so Hoeffding’s inequality applies to squared loss.” Hoeffding requires bounded observations; finite variance does not bound \((Y-g(X))^2\).
• Correction: Verify a bounded-loss condition, use a concentration result with appropriate tail assumptions, or truncate and retain the tail error explicitly.
• Common mistake: “Clipping the responses proves a bound for the original risk with no extra term.” Clipping changes the population risk even when it happens rarely.
• Correction: Account separately for the empirical event of observing a tail value and the expected population loss contributed by the tail.

For neural regression, the practical implication is to state the response or noise assumption and the prediction constraint alongside any squared-loss generalization claim. A bound whose tail premise is not justified can fail precisely on the observations that dominate prediction risk.