Regularization Scaling for Kernel Classification Objectives

Why Objective Scaling Matters

A regularized kernel classifier balances two quantities: empirical loss on the observed data and a penalty on the size of the score function. The balance depends not only on the numerical value called the regularization parameter, but also on whether the empirical losses are added or averaged. These two objective conventions are both common. Their coefficients are not interchangeable without a sample-size adjustment.

Let $(X_i,Y_i)$, $i=1,\ldots,n$, be the observed training sample, with $Y_i\in\{-1,+1\}$. Let $\mathcal H$ be a reproducing kernel Hilbert space (RKHS), with norm $\|\cdot\|_{\mathcal H}$, and let $\phi:\mathbb R\to[0,\infty)$ be a loss applied to the margin $Y_i f(X_i)$. Examples include hinge loss and logistic loss. The scaling argument below does not depend on which of these losses is used.

Definition: For a penalty coefficient $\lambda_{\mathrm{sum}}>0$, the summed-loss objective is $$ J_{\mathrm{sum}}(f;\lambda_{\mathrm{sum}}) = \sum_{i=1}^n \phi\bigl(Y_i f(X_i)\bigr) +\frac{\lambda_{\mathrm{sum}}}{2}\|f\|_{\mathcal H}^2. $$ For a coefficient $\lambda_{\mathrm{avg}}>0$, the averaged-loss objective is $$ J_{\mathrm{avg}}(f;\lambda_{\mathrm{avg}}) = \frac{1}{n}\sum_{i=1}^n \phi\bigl(Y_i f(X_i)\bigr) +\frac{\lambda_{\mathrm{avg}}}{2}\|f\|_{\mathcal H}^2. $$ The factor $1/2$ is a convention that simplifies some derivatives. It does not change the rescaling when it is used in both objectives.

The key comparison is between the loss and penalty relative to each other. Dividing an entire objective by a positive number does not change its minimizers, but dividing only the loss term does change that relative balance unless the penalty coefficient is adjusted too.

The Exact Conversion

The summed and averaged conventions represent the same optimization problem when their coefficients are related by the sample size. This is an algebraic identity, so it holds for any fixed sample and any real-valued loss for which the objectives are defined.

Theorem: For every $f\in\mathcal H$, $$ \frac{1}{n}J_{\mathrm{sum}}(f;\lambda_{\mathrm{sum}}) = J_{\mathrm{avg}}\left(f;\frac{\lambda_{\mathrm{sum}}}{n}\right). $$ Consequently, their sets of minimizers are identical when $$ \lambda_{\mathrm{avg}}=\frac{\lambda_{\mathrm{sum}}}{n}, \qquad\text{or equivalently}\qquad \lambda_{\mathrm{sum}}=n\lambda_{\mathrm{avg}}. $$

Proof. Divide the summed objective by $n$, which is positive:

\begin{aligned} \frac{1}{n}J_{\mathrm{sum}}(f;\lambda_{\mathrm{sum}}) &=\frac{1}{n}\sum_{i=1}^n\phi\bigl(Y_i f(X_i)\bigr) +\frac{\lambda_{\mathrm{sum}}}{2n}\|f\|_{\mathcal H}^2\\ &=J_{\mathrm{avg}}\left(f;\frac{\lambda_{\mathrm{sum}}}{n}\right). \end{aligned}

For any two functions $f,g$, division by a positive constant preserves the ordering of their objective values. Thus $f$ minimizes the summed objective if and only if it minimizes the divided objective, which is the averaged objective with coefficient $\lambda_{\mathrm{sum}}/n$. This proves equality of the minimizer sets, including if a minimizer is not attained. $\square$

This result does not say that a single numerical value of $\lambda$ works identically under both conventions. It says that a specific pair of values, differing by a factor of $n$, yields the same objective up to a positive multiplier and therefore the same minimizers.

Worked Example: Converting Coefficients and Checking Objective Values

Suppose $n=80$ and the summed-loss coefficient is $\lambda_{\mathrm{sum}}=24$. The matching averaged-loss coefficient is $$ \lambda_{\mathrm{avg}}=\frac{24}{80}=0.3. $$ For a candidate function $f$, suppose the sum of its 80 losses is 12 and $\|f\|_{\mathcal H}^2=9$. Then

J_{\mathrm{sum}}(f;24)=12+\frac{24}{2}(9)=12+108=120.

The averaged objective at the converted coefficient is $$ J_{\mathrm{avg}}(f;0.3)=\frac{12}{80}+\frac{0.3}{2}(9)=0.15+1.35=1.5. $$ Since $120/80=1.5$, the averaged objective is exactly the summed objective divided by 80. The calculation applies to every candidate $f$, not just this one, so the minimizers agree.

A General Coefficient Rule

The same reasoning helps when an objective is written with different constants in front of its two terms. Suppose an objective has the form $a\sum_i\phi(Y_i f(X_i))+b\|f\|_{\mathcal H}^2$, where $a>0$ and $b\geq 0$. Dividing by $an$ gives average loss plus a norm penalty. In the convention that writes the penalty as $\lambda\|f\|_{\mathcal H}^2/2$, its effective averaged coefficient is $2b/(an)$.

Result: For $a>0$, $$ a\sum_{i=1}^n\phi\bigl(Y_i f(X_i)\bigr) +b\|f\|_{\mathcal H}^2 $$ has the same minimizers as $$ \frac{1}{n}\sum_{i=1}^n\phi\bigl(Y_i f(X_i)\bigr) +\frac{\lambda_{\mathrm{eff}}}{2}\|f\|_{\mathcal H}^2, \qquad \lambda_{\mathrm{eff}}=\frac{2b}{an}. $$

Derivation. Divide the first objective by $an>0$. Its loss term becomes the average loss, and its penalty term becomes $\{b/(an)\}\|f\|_{\mathcal H}^2$. Equating this penalty coefficient with $\lambda_{\mathrm{eff}}/2$ gives $\lambda_{\mathrm{eff}}=2b/(an)$. Positive rescaling preserves minimizers, as in the theorem above. $\square$

In particular, if a source writes the penalty as $\lambda\|f\|_{\mathcal H}^2$ rather than $\lambda\|f\|_{\mathcal H}^2/2$, then the coefficient called $\lambda$ is half the coefficient in the convention used here. Always compare the complete objective, including its constants, rather than comparing parameter names alone.

Worked Example: A Hinge-Loss Classifier with a Constant Kernel

Take the constant kernel $K(x,x')=1$, and suppose all $n$ labels are $+1$. Its RKHS consists of constant functions $f(x)=t$, with $\|f\|_{\mathcal H}^2=t^2$. Under hinge loss $\phi(u)=\max(0,1-u)$, the summed objective is $$ J_{\mathrm{sum}}(t;\lambda_{\mathrm{sum}}) =n\max(0,1-t)+\frac{\lambda_{\mathrm{sum}}}{2}t^2. $$ For $t<1$, its derivative is $-n+\lambda_{\mathrm{sum}}t$. The stationary point is $t=n/\lambda_{\mathrm{sum}}$. If this is below 1, the objective decreases before that point and increases after it; if it is at least 1, the objective decreases up to 1 and then the penalty increases. Thus a minimizer is $$ t_{\mathrm{sum}}^*=\min\left(1,\frac{n}{\lambda_{\mathrm{sum}}}\right). $$

For the averaged objective, the same calculation, with $n$ replaced by 1 in the loss term, gives $$ t_{\mathrm{avg}}^*=\min\left(1,\frac{1}{\lambda_{\mathrm{avg}}}\right). $$ With $n=20$ and $\lambda_{\mathrm{sum}}=40$, the summed objective gives $t_{\mathrm{sum}}^*=\min(1,20/40)=0.5$. The matching averaged coefficient is $40/20=2$, which gives $t_{\mathrm{avg}}^*=\min(1,1/2)=0.5$. The explicit minimizers confirm the general conversion.

What Happens When Sample Size Changes?

The conversion rule has a direct implication for comparing fits across sample sizes. Holding $\lambda_{\mathrm{sum}}$ fixed while increasing $n$ makes its equivalent averaged coefficient $\lambda_{\mathrm{sum}}/n$ smaller. Holding $\lambda_{\mathrm{avg}}$ fixed instead requires the summed coefficient to grow in proportion to $n$. These are statements about the objective's relative penalty, not guarantees about a classifier's predictive performance.

Takeaway: In the summed-loss convention, a fixed penalty coefficient corresponds to an averaged-loss penalty that decreases as $1/n$. In the averaged-loss convention, a fixed penalty coefficient corresponds to a summed-loss penalty that increases as $n$.

Worked Example: Fixed Coefficient, Different Sample Sizes

Consider an averaged-loss objective with $\lambda_{\mathrm{avg}}=0.2$. At sample size $n=25$, the equivalent summed coefficient is $25(0.2)=5$. At $n=100$, it is $100(0.2)=20$. Keeping the averaged coefficient fixed therefore does not mean keeping the summed coefficient fixed.

Now reverse the comparison. If $\lambda_{\mathrm{sum}}=5$ is held fixed, the equivalent averaged coefficients are $5/25=0.2$ at $n=25$ and $5/100=0.05$ at $n=100$. The penalty relative to average loss is four times smaller at $n=100$ than at $n=25$.

The constant-kernel hinge example makes this change visible in fitted scores. With $\lambda_{\mathrm{sum}}=20$, at $n=5$ its minimizer is $t^*=\min(1,5/20)=0.25$; at $n=20$ it is $t^*=\min(1,20/20)=1$. The coefficient is unchanged, but the summed loss has more weight relative to the penalty as the sample grows. This example isolates the objective-scaling effect; it does not claim that this toy model represents the behavior of every kernel classifier.

Common Pitfall: Reusing the Same Number

A frequent mistake is to see the same loss and penalty in two formulas, substitute the same numerical $\lambda$, and conclude that the resulting classifier is unchanged when one formula sums losses and the other averages them. That reasoning overlooks the factor of $n$. For the objectives defined here, using the same positive number in both conventions gives matching minimizers only in the special case $n=1$, apart from possible coincidences caused by a particular data set or loss.

For instance, with $n=20$ and numerical coefficient 2, the constant-kernel hinge example gives $t^*_{\mathrm{sum}}=\min(1,20/2)=1$, while the averaged objective gives $t^*_{\mathrm{avg}}=\min(1,1/2)=0.5$. The correction is to translate the coefficient: $\lambda_{\mathrm{avg}}=\lambda_{\mathrm{sum}}/20$, or $\lambda_{\mathrm{sum}}=20\lambda_{\mathrm{avg}}$, depending on which objective is the reference.

When reading or reporting a regularized classification fit, record the objective itself, not just a parameter label. State whether losses are summed or averaged, how the norm penalty is written, and whether any additional constants multiply either term. Only then can coefficients be compared, reproduced, or interpreted across sample sizes.

Check Your Understanding

Use the stated objective conventions when answering each question.

If $n=60$ and $\lambda_{\mathrm{sum}}=12$, what value of $\lambda_{\mathrm{avg}}$ gives the same minimizers?
Why does dividing the entire summed objective by $n$ preserve its minimizers?
For a summed-loss objective with fixed $\lambda_{\mathrm{sum}}$, how does its equivalent averaged-loss coefficient change as $n$ increases?
An objective is written as $\sum_i\phi(Y_i f(X_i))+c\|f\|_{\mathcal H}^2$. What is its equivalent averaged-loss coefficient under the convention $\lambda\|f\|_{\mathcal H}^2/2$?
Why is it generally incorrect to use the same numerical regularization coefficient in a summed-loss and an averaged-loss objective?