Why Separate the Sources of Risk?
A neural network can predict poorly for several distinct reasons. The chosen architecture may not represent a good predictor; the finite training sample may make an empirically attractive predictor look better than it is; or the training algorithm may fail to minimize even the empirical objective. These are different limitations, and improving one does not automatically fix the others.
The three-term decomposition makes that distinction precise. It is an identity first, not a rate theorem: by itself it says nothing about how small the terms are. To obtain numerical or asymptotic bounds, one needs additional assumptions about the loss, the network class, the data, and often the training procedure.
Risk, Network Class, and Comparators
Let \(Z=(X,Y)\) have distribution \(P\), and let \(Z_1,\ldots,Z_n\) be independent observations with this distribution. For a measurable predictor \(f\) and a nonnegative measurable loss \(\ell\), define its population risk and empirical risk by
Consider a fixed neural-network architecture and parameter constraints, which together define a class of predictors \(\mathcal F_N\). The subscript emphasizes the network class; it does not mean the class changes with sample size unless that is explicitly specified. Let \(\hat f\in\mathcal F_N\) be the output of training, and let
where \(\mathcal G\) is a stated benchmark class of measurable predictors containing \(\mathcal F_N\). Often \(R^*\) is the Bayes risk, the smallest risk among all measurable predictors under the chosen loss. The comparator class must be stated: excess risk relative to a restricted benchmark is not the same quantity as excess risk relative to the Bayes risk.
The assumptions in this definition matter. Risks must be defined, so the loss should be measurable and its expectation should exist; a convenient sufficient condition is finite risk for the predictors being compared. Nonnegative loss permits extended-valued risks, but the finite-risk statements below assume the displayed risks are finite. The approximation error is nonnegative when \(\mathcal G\) contains \(\mathcal F_N\). The optimization error is nonnegative by the definition of an infimum. The estimator must belong to \(\mathcal F_N\); otherwise this particular optimization gap does not describe its training objective.
The Exact Three-Term Identity
Proof. Add and subtract \(R(f_N)\), \(R_n(\hat f)\), \(m_n\), and \(R_n(f_N)\). Group the resulting differences as follows:
The estimation fluctuation \(E_n\) is signed: it need not be nonnegative. In particular, the term \(m_n-R_n(f_N)\) is nonpositive because \(f_N\in\mathcal F_N\). It would be misleading to call every term in the exact identity a nonnegative error. The useful nonnegative estimation quantity usually comes from bounding the signed fluctuation from above.
Worked Example: Approximation Error for Constant Predictors
Let \(X\) be uniform on \([0,1]\), let \(Y=X^2+\varepsilon\), and suppose \(\varepsilon\) is independent of \(X\), has mean zero, and has variance \(1\). Use squared-error loss \(\ell(y,a)=(y-a)^2\). Compare the benchmark of all measurable predictors with the network class consisting only of constant predictors \(f_c(x)=c\).
For squared loss, the conditional mean minimizes conditional expected loss, so the benchmark predictor is \(f^*(x)=E[Y\mid X=x]=x^2\). Its risk is \(R^*=E[\varepsilon^2]=1\). For a constant \(c\), independence and \(E[\varepsilon]=0\) give
Expanding the first expectation gives \(E[X^4]-2cE[X^2]+c^2\). Since \(E[X^2]=1/3\) and \(E[X^4]=1/5\), this is minimized at \(c=1/3\). The resulting risk is
Thus \(f_N=f_{1/3}\), and the approximation error is \(A_N=4/45\). This error remains even with unlimited data and perfect optimization within this constant-only class: the class cannot represent the varying conditional mean \(x^2\).
Bounding the Estimation Fluctuation
A common way to control the signed estimation term is to compare population and empirical risks uniformly over the network class. Define
For this supremum to be a well-defined random variable, assume, for example, that \(\mathcal F_N\) is countable and each empirical risk is measurable. More general classes can be used when an appropriate measurability or separability condition ensures the supremum is measurable. Assume also that the risks are finite.
Proof. Start with the exact identity. Since \(f_N\in\mathcal F_N\), the definition of the empirical infimum gives \(m_n-R_n(f_N)\leq 0\). The other two parts of \(E_n\) are each at most \(U_n\):
Therefore \(E_n\leq U_n+0+U_n=2U_n\). Substituting this inequality into \(R(\hat f)-R^*=A_N+E_n+\delta_n\) proves the result. \(\square\)
The bound separates the roles of the three terms. \(A_N\) concerns what the architecture can represent; \(U_n\) concerns how well empirical risks track population risks across the whole class; and \(\delta_n\) concerns how closely training approaches the empirical infimum. This is an upper bound, not a claim that each source contributes exactly that much risk. In particular, a small optimization gap cannot compensate for a large approximation error.
Worked Example: Sampling Fluctuation Can Have Either Sign
Let \(X\) be irrelevant, \(Y\sim\mathrm{Bernoulli}(0.4)\), and use squared loss with the neural class of constant outputs \(f_c(x)=c\), \(0\leq c\leq1\). The benchmark allows all measurable predictors. The population-optimal constant is \(c=0.4\), which is also the conditional mean, so \(R^*=R(f_N)=0.4(0.6)^2+0.6(0.4)^2=0.24\), and \(A_N=0\).
Suppose the sample is \(Y_1,Y_2,Y_3=(0,1,1)\). The empirical squared risk is minimized at the sample mean \(2/3\), and
Take the trained output to be \(\hat f=f_{1/2}\). Its empirical risk is \(R_n(\hat f)=1/4\), so \(\delta_n=1/4-2/9=1/36\). Its population risk is \(R(\hat f)=0.4(1/2)^2+0.6(1/2)^2=1/4\); hence its excess risk is \(1/4-0.24=1/100\).
The exact estimation fluctuation is
Here \(R_n(f_N)=R_n(f_{0.4})=22/75\). The identity checks: \(A_N+E_n+\delta_n=0-4/225+1/36=1/100\). This sample makes the signed estimation term negative; the empirical risks happen to favor a predictor whose true risk is not worse than the population-optimal class risk.
Imperfect Optimization and What the Terms Do Not Say
For a neural network, the empirical optimization gap is measured relative to the infimum over the specified class, not relative to the output of a particular optimizer or initialization. It is nonnegative even when the infimum is not attained. If the algorithm returns an approximate empirical minimizer, the gap records that shortfall. If it returns a global empirical minimizer, then \(\delta_n=0\), but \(A_N\) and the sampling fluctuation remain.
Worked Example: A Network That Is Not Empirically Optimal
Use the Bernoulli sample \(0,1,1\) and squared loss from the preceding example. A network with a constant output bias can realize every constant \(c\in[0,1]\); take this as the class. Its empirical risk is minimized at \(c=2/3\), with \(m_n=2/9\). Suppose training instead returns the constant-output network \(\hat f=f_{1/2}\), whose empirical risk is \(1/4\). The optimization gap is
The class has zero approximation error relative to all measurable predictors in this example, because the conditional mean is the constant \(0.4\), which the class represents. The returned network has population risk \(1/4\), so its excess risk is \(1/100\). Here the exact estimation fluctuation is \(-4/225\), as calculated above, and the identity gives \(0-4/225+1/36=1/100\). The positive optimization gap is therefore not itself equal to the excess risk; the other terms are essential to the accounting.
The bound involving \(U_n\) is useful only if \(U_n\) can be controlled. A finite-sample guarantee requires assumptions that make this uniform deviation small, such as suitable restrictions on the class and loss. Merely naming a neural architecture does not establish such control. Nor does the identity require that the loss be squared error: it applies to any nonnegative measurable loss for which the risks and displayed quantities are finite.
Common Pitfall
- Wrong reasoning: “The optimizer found a network with almost minimal training loss, so its population risk must be close to optimal.” A small \(\delta_n\) only compares training risk within \(\mathcal F_N\). It does not show that empirical and population risks are close, or that the class can approximate a good predictor.
- Correction: Check all three contributions. Approximation requires a suitable class relative to a stated benchmark; estimation requires control of sampling fluctuation; optimization requires a small empirical gap. The risk bound makes these separate requirements explicit.
- Another pitfall: Treating the exact estimation fluctuation \(E_n\) as nonnegative. It is signed; only the upper bound \(E_n\leq 2U_n\) is guaranteed here.
Check Your Understanding
Use the definitions and assumptions in the tutorial to answer these questions.
- Why must the benchmark class be stated when defining approximation error?
- Which two terms in the exact decomposition are necessarily nonnegative, and why can the estimation fluctuation have either sign?
- What does \(\delta_n=0\) establish, and what does it leave unresolved?
- In the constant-predictor example with \(Y=X^2+\varepsilon\), what feature of the network class produces the approximation error?
- Which measurability issue arises when defining \(U_n\) for an uncountable network class?