A Worked Squared-Loss Bound for Neural Regression

Why Separate the Sources of Error?

A neural regression predictor can perform poorly for different reasons. The chosen network family may not approximate the regression function well; finite data may make empirical risk a noisy guide to population risk; or the training algorithm may not find a sufficiently good empirical solution. A useful finite-sample guarantee keeps these contributions visible rather than combining them into one unexplained error term.

We consider a finite, prespecified library of bounded neural predictors. This setting gives a direct and transparent squared-loss bound: the estimation term depends on the size of the library, while the approximation and optimization terms retain their separate meanings. A continuous neural family is not finite merely because it has finitely many parameters; the finite-library result below applies directly to a finite candidate set, such as a fixed collection of trained or discretized predictors.

Setup and Squared-Risk Identity

Let \((X_i,Y_i)\), \(i=1,\ldots,n\), be independent and identically distributed copies of \((X,Y)\). Assume \(|Y|\leq B\) almost surely for a fixed \(B>0\). Let \(\mathcal F\) be a nonempty finite set of measurable neural predictors, fixed independently of the sample, satisfying \(|f(x)|\leq B\) for every \(f\in\mathcal F\) and every input \(x\). Define the population and empirical squared risks by

Write \(\eta(x)=E[Y\mid X=x]\), a version of the conditional mean. Since \(Y\) is bounded, \(|\eta(X)|\leq B\) almost surely. The conditional mean minimizes squared prediction risk pointwise in \(X\), as the following identity shows.

To verify the identity, condition on \(X\) and write \(Y-f(X)=(Y-\eta(X))+(\eta(X)-f(X))\). Expanding the square gives a squared residual, a squared prediction difference, and a cross term. The conditional expectation of that cross term is zero because \(E[Y-\eta(X)\mid X]=0\). Thus

Taking expectations proves the identity. In particular, the best population risk available in \(\mathcal F\) exceeds the regression-function risk by the approximation error \(A_{\mathcal F}=\min_{f\in\mathcal F}E[(f(X)-\eta(X))^2]\). The minimum exists because \(\mathcal F\) is finite.

A Finite-Library Estimation Bound

For any fixed \(f\in\mathcal F\), the squared loss lies in \([0,4B^2]\): both \(Y\) and \(f(X)\) lie in \([-B,B]\). Hoeffding's inequality therefore controls the deviation between its empirical and population risks. A union bound makes that control simultaneous over the whole finite library.

Proof. Fix \(f\in\mathcal F\). Its \(n\) observed losses are independent, identically distributed random variables in an interval of length \(4B^2\), with mean \(R(f)\). Hoeffding's inequality gives, for every \(t>0\),

The probability that this deviation exceeds \(t\) for at least one of the \(M\) predictors is at most the sum of their individual probabilities, hence at most \(2M\exp(-nt^2/(8B^4))\). Substituting \(t=4B^2\sqrt{\log(2M/\delta)/(2n)}\) makes that upper bound equal to \(\delta\). This proves the simultaneous statement. \(\square\)

The Excess-Risk Bound With Optimization Error

Let \(\widehat f\in\mathcal F\) be the predictor returned by training. It need not minimize empirical risk exactly. Assume that it satisfies the empirical optimization condition \(\widehat R_n(\widehat f)\leq\min_{f\in\mathcal F}\widehat R_n(f)+\varepsilon\), for some \(\varepsilon\geq0\). The condition may hold deterministically or on the event being considered.

Proof. Work on the event in the finite-library theorem, and choose \(f^*\in\mathcal F\) minimizing population risk. The uniform deviation bound applies to both the returned predictor and \(f^*\). Using the empirical optimization condition between them gives

By the squared-risk identity and the definition of \(A_{\mathcal F}\), \(R(f^*)-R(\eta)=A_{\mathcal F}\). Subtracting \(R(\eta)\) in the preceding inequality and substituting the value of \(2t_{n,\delta}\) proves the bound. \(\square\)

The three terms describe distinct limitations. The first is determined by the predictor family and the distribution of \(X\); more data alone does not remove it. The second records how far training stops from the best empirical candidate. The last is the cost of using a sample to compare \(M\) bounded candidates, and decreases as \(n\) grows for fixed \(M\), \(B\), and \(\delta\).

Interpretation and a Common Pitfall

The finite-library result is useful when the candidate set is fixed before observing the training sample and every candidate's output is bounded. For a neural workflow, the library might consist of a prespecified list of architectures and trained candidates. A bound for a continuously parameterized family requires a suitable complexity or uniform-deviation argument; it does not follow by treating the number of weights as the number \(M\) of predictors.

A common mistake is to compare \(\widehat R_n(\widehat f)\) directly with \(R(\widehat f)\) as though the empirical risk were an unbiased estimate uniformly over every predictor selected after seeing the data. For each fixed predictor its empirical risk is unbiased, but that fact alone does not control a data-dependent choice. The correction is to establish a simultaneous deviation bound over the candidate set, then use it for both the selected predictor and the comparison predictor, as in the proof. Another mistake is to omit the optimization gap: if training returns a predictor above the empirical minimum, the proof must pay for that gap through \(\varepsilon\).

Finally, the bound is not automatically informative just because it is mathematically valid. If the estimation term is large, the guarantee may be loose even when the realized predictor performs well. Its purpose is to expose the assumptions and the separate sources of risk, not to promise a sharp numerical prediction in every sample size regime.