Deriving The Optimal Model Scale

Balancing Approximation and Estimation

For a sieve or other indexed family of models, increasing the model index can make the best available approximation more accurate, while also making estimation harder. The oracle decomposition for sieve risk separates these two contributions. Here we study the scale that minimizes a particular upper bound on their sum; we do not assume that minimizing the bound identifies a uniquely correct model.

Suppose the approximation contribution decreases as a power of the model index $m$, and the estimation contribution increases as a power of $m/n$, where $n$ is sample size. Positive constants determine their magnitudes, and positive exponents determine how quickly they change. The goal is to choose $m$ so that neither contribution is needlessly dominant.

Definition: For constants $A>0$, $D>0$, $\alpha>0$, $\beta>0$, and sample size $n>0$, define the polynomial bound $$ B_n(m)=A m^{-\alpha}+D\left(\frac{m}{n}\right)^\beta,\qquad m>0. $$ The first term is the approximation contribution; the second is the estimation contribution. In an actual model-selection problem, $m$ is usually a positive integer in an admissible set. We initially allow real $m>0$ to find the continuous optimum.

The exponents need not match. For example, an approximation error proportional to $m^{-2}$ paired with an estimation error proportional to $\sqrt{m/n}$ has $\alpha=2$ and $\beta=1/2$. The optimal scale depends on both exponents and on the constants $A$ and $D$, not only on the sample size.

The Continuous Optimal Scale

Differentiating the bound shows where it stops decreasing and begins increasing. The derivative is $$ B_n'(m)=-A\alpha m^{-\alpha-1}+D\beta n^{-\beta}m^{\beta-1}. $$ Setting it to zero gives $$ m^{\alpha+\beta}=\frac{A\alpha}{D\beta}n^\beta. $$ This suggests the following result.

Theorem: For $A,D,\alpha,\beta,n>0$, the function $B_n(m)$ has a unique global minimum over real $m>0$, attained at $$ m_*=\left(\frac{A\alpha}{D\beta}\right)^{1/(\alpha+\beta)} n^{\beta/(\alpha+\beta)}. $$ At this minimum, the two contributions satisfy $$ D\left(\frac{m_*}{n}\right)^\beta =\frac{\alpha}{\beta}A m_*^{-\alpha}. $$

Proof. Multiplying the derivative by the positive quantity $m^{\alpha+1}$ gives $$ m^{\alpha+1}B_n'(m)=-A\alpha+D\beta n^{-\beta}m^{\alpha+\beta}. $$ The right-hand side is strictly increasing in $m>0$, because $\alpha+\beta>0$. It is negative for sufficiently small $m$ and positive for sufficiently large $m$, so it has exactly one zero. Solving for that zero gives the displayed formula for $m_*$. Since the sign of $B_n'(m)$ is the sign of the right-hand side, $B_n$ decreases before $m_*$ and increases after it. Thus $m_*$ is the unique global minimizer. Substituting $B_n'(m_*)=0$ and rearranging gives the stated relation between the two terms. $\square$

The relation at the optimum is a weighted balance, not necessarily equality of the two terms. If $\alpha=\beta$, the two contributions are equal. If the exponents differ, their ratio is $\alpha/\beta$: a steeper approximation decrease, for example, changes the balance point.

Worked Example: Deriving a Sample-Size Scale

Suppose a risk bound has approximation term $4m^{-2}$ and estimation term $\sqrt{m/n}$. Thus $A=4$, $\alpha=2$, $D=1$, and $\beta=1/2$. Find the continuous optimal scale as a function of $n$.

The formula gives $$ m_*=\left(\frac{4\cdot 2}{1\cdot(1/2)}\right)^{1/(2+1/2)} n^{(1/2)/(2+1/2)} =16^{2/5}n^{1/5}. $$ Since $16^{2/5}\approx 3.031$, the optimal scale grows approximately as $3.031n^{1/5}$.

The approximation contribution at this scale is proportional to $m_*^{-2}$, hence to $n^{-2/5}$. The estimation contribution is proportional to $(m_*/n)^{1/2}$, which also scales as $n^{-2/5}$. The constants differ according to the weighted balance: here the estimation contribution is $2/(1/2)=4$ times the approximation contribution at the continuous optimum.

The Minimized Bound and Its Rate

The optimal index is useful because it tells us not just which model scale to consider, but how fast the resulting bound can decrease with sample size. Let $$ K=\left(\frac{A\alpha}{D\beta}\right)^{1/(\alpha+\beta)}, $$ so that $m_*=Kn^{\beta/(\alpha+\beta)}$. Substituting this expression into each term yields the same power of $n$.

Result: The minimum continuous bound is $$ B_n(m_*)= \left(AK^{-\alpha}+DK^\beta\right) n^{-\alpha\beta/(\alpha+\beta)}, \qquad K=\left(\frac{A\alpha}{D\beta}\right)^{1/(\alpha+\beta)}. $$ Equivalently, $B_n(m_*)=\frac{\alpha+\beta}{\beta}A m_*^{-\alpha}$.

Derivation. From $m_*=Kn^{\beta/(\alpha+\beta)}$, the approximation term is $$ A m_*^{-\alpha}=AK^{-\alpha}n^{-\alpha\beta/(\alpha+\beta)}. $$ For the estimation term, $$ D\left(\frac{m_*}{n}\right)^\beta =DK^\beta n^{\beta(\beta/(\alpha+\beta)-1)} =DK^\beta n^{-\alpha\beta/(\alpha+\beta)}. $$ Adding these expressions proves the first formula. The second follows from the balance relation in the theorem: the estimation term is $(\alpha/\beta)$ times the approximation term, so their sum is $((\alpha+\beta)/\beta)A m_*^{-\alpha}$. $\square$

The exponent $\alpha\beta/(\alpha+\beta)$ describes the rate of this optimized bound. It is determined by the two power laws: a faster decrease in approximation error or a slower increase in estimation error changes the attainable rate. This is a rate for the stated bound under its assumptions, not a guarantee that every learning problem has exactly this risk rate.

Worked Example: Checking the Minimized Rate

Return to $B_n(m)=4m^{-2}+\sqrt{m/n}$. We found $m_*=16^{2/5}n^{1/5}$. The general rate exponent is $$ \frac{\alpha\beta}{\alpha+\beta} =\frac{2(1/2)}{2+1/2} =\frac{2}{5}. $$ Thus the optimized bound has order $n^{-2/5}$. This agrees with direct substitution: $m_*^{-2}$ has order $n^{-2/5}$, not $n^{-1/5}$—so it is important to check the calculation carefully.

Here the correct substitution gives $m_*^{-2}$ of order $n^{-2/5}$, while $\sqrt{m_*/n}$ has order $n^{-2/5}$ as well. Correspondingly, the rate exponent from the formula is $$ \frac{2(1/2)}{2+1/2}=\frac{1}{2.5}=\frac{2}{5}. $$ Therefore the minimized bound has order $n^{-2/5}$, consistent with both terms. The displayed fraction simplifies to $2/5$, since its numerator is $1$ and its denominator is $5/2$.

Choosing an Integer Model Index

In practice, a model index is usually an integer. When every positive integer is admissible and $m_*\geq 1$, the continuous monotonicity result implies that an integer minimizer can be found by comparing the admissible integers immediately below and above $m_*$. If $m_*$ is itself an integer, it is the integer minimizer. Otherwise, evaluate $B_n$ at $\lfloor m_*\rfloor$ and $\lceil m_*\rceil$ and select the smaller value.

There is an important boundary case. If $0<m_*<1$, then $B_n$ is increasing for every $m\geq1$, because all such indices lie to the right of its continuous minimum. The positive-integer minimizer is therefore $m=1$; comparing the floor of $m_*$ would be invalid, since that floor is zero and the bound is undefined there. For a restricted contiguous set of admissible integers, check the neighboring admissible indices when $m_*$ lies inside the range, and check the appropriate endpoint when it lies outside.

Worked Example: Comparing the Two Neighboring Integers

Consider $$ B_{10000}(m)=2m^{-3/4}+0.03\sqrt{m}, $$ which has $A=2$, $\alpha=3/4$, $D=3$, $\beta=1/2$, and $n=10000$, because $3\sqrt{m/10000}=0.03\sqrt m$. The continuous optimum is $$ m_*=\left(\frac{2(3/4)}{3(1/2)}\right)^{1/(3/4+1/2)} 10000^{(1/2)/(3/4+1/2)} =100^{4/5}\approx39.81. $$ The neighboring positive integers are 39 and 40.

Direct substitution gives $$ B_{10000}(39)=2(39^{-3/4})+0.03\sqrt{39}\approx0.31550376\approx0.31550, $$ and $$ B_{10000}(40)=2(40^{-3/4})+0.03\sqrt{40}\approx0.31548. $$ Since $B_{10000}(40)<B_{10000}(39)$, the better integer choice is $m=40$. The continuous optimum locates the nearby candidates; it does not by itself decide between them.

When the Admissible Range Has a Boundary

Some problems only allow indices in a finite range, for example $1\leq m\leq n$. The continuous optimum remains informative, but the selected scale must be feasible. If $m_*$ is below the range, the bound is increasing throughout the range and its minimum is at the lower endpoint. If $m_*$ is above the range, the bound is decreasing throughout the range and its minimum is at the upper endpoint. If it lies inside, compare the adjacent admissible integers around it.

Worked Example: An Optimum Beyond the Allowed Range

Let $n=100$, $A=100$, $\alpha=1$, $D=1$, and $\beta=1/2$, with admissible integer indices $1\leq m\leq100$. The bound is $$ B_{100}(m)=\frac{100}{m}+\sqrt{\frac{m}{100}}. $$ Its continuous optimum is $$ m_*=\left(\frac{100}{1/2}\right)^{1/(1+1/2)} 100^{(1/2)/(1+1/2)} =200^{2/3}100^{1/3}=2000^{2/3}\approx158.74. $$ This calculation must be read carefully: using the formula directly, the two factors are $200^{2/3}$ and $100^{1/3}$, because the factor $n^\beta$ is already represented in the numerator of the general formula. Thus $$ m_*=\left(\frac{100}{1/2}\,100^{1/2}\right)^{2/3} =2000^{2/3}\approx158.74. $$ So the continuous optimum exceeds the allowed maximum. The derivative is negative for every $m<158.74$, and hence the bound decreases throughout the allowed interval. The best admissible index is $m=100$, where $$ B_{100}(100)=\frac{100}{100}+\sqrt{\frac{100}{100}}=1+1=2. $$

Common Pitfall: Balancing Without Checking the Domain

A common mistake is to solve the derivative equation and treat its solution as the model choice, regardless of whether the solution is an allowed index. Another is to assume that the two error terms must be equal. The derivative equation instead gives the weighted relation $D(m_*/n)^\beta=(\alpha/\beta)A m_*^{-\alpha}$, and the final choice still depends on the admissible range and integer constraint. In the boundary example, the continuous optimum is not admissible; the correct choice is the endpoint where the bound is smallest.

The calculation also concerns a bound, not a direct optimization of the unknown true risk. Constants in a bound may be conservative or unknown in applications. The derivation explains the scale implied by the bound and its polynomial assumptions; it does not justify treating those constants as exactly known or claiming that the resulting index is universally optimal.

Key takeaway: For a bound $A m^{-\alpha}+D(m/n)^\beta$, the continuous balancing scale is proportional to $n^{\beta/(\alpha+\beta)}$, and the minimized bound is proportional to $n^{-\alpha\beta/(\alpha+\beta)}$. Convert that scale to a model choice by respecting the integer and admissibility constraints.

Check Your Understanding

Use the polynomial bound and domain rules developed above.

For $A,D,\alpha,\beta>0$, write the continuous minimizing index for $A m^{-\alpha}+D(m/n)^\beta$.
At the continuous optimum, what is the ratio of the estimation term to the approximation term? Under what condition are they equal?
For $\alpha=1$ and $\beta=1$, what are the powers of $n$ in the optimal index and the minimized bound?
If the continuous optimum is $m_*=0.6$ and all positive integers are allowed, which integer minimizes the bound? Explain why comparing with zero is not appropriate.
If a continuous optimum lies above the largest admissible index, which endpoint minimizes the bound over a contiguous admissible range, and why?