Balancing Fit Against Model Size

Why Model Choice Needs More Than a Fit Measure

For a fixed set of observations, a regression subset with more predictors can often fit the training data at least as closely as a smaller subset. But a smaller residual sum of squares does not, by itself, establish that the larger model is the better choice. A useful comparison must reward fit while also accounting for how many predictors the model uses.

This tutorial develops a generic score for making that tradeoff. It is a framework rather than a named information criterion: the penalty is stated explicitly, and the preferred candidate is the one with the lowest score. The exact choice of penalty matters, so a score is meaningful only when its formula, units and candidate models are clear.

Definition: For a candidate regression model $M$, let $RSS(M)=\sum_{i=1}^{n}(y_i-\hat{y}_{i,M})^2$ be its residual sum of squares, and let $k(M)$ be its number of predictors. For a chosen penalty $\lambda\geq 0$, define the generic penalized score $$ J_\lambda(M)=RSS(M)+\lambda k(M). $$ Among the candidates being compared, prefer the model with the smallest $J_\lambda(M)$. Here the intercept is included in every model and is not counted in $k(M)$.

This score is intended for comparisons that use the same response variable, observations and fitting procedure. The examples use ordinary least squares (OLS) with an intercept. A different analysis sample can change the residual sum of squares for reasons unrelated to subset size, making this comparison unfair. In the examples, each added predictor contributes one unit to $k$; if candidate models use predictors with different numbers of parameters, the complexity measure must reflect that difference.

The penalty $\lambda$ must be nonnegative and expressed in the same units as $RSS$. If $RSS$ is measured in squared thousands of dollars, then $\lambda$ must also be in squared thousands of dollars per counted predictor. The score is not a probability or a test statistic: it is a defined rule for ranking the candidate models.

How Fit and Complexity Compete

A smaller $RSS$ rewards closer training fit; the term $\lambda k(M)$ charges for model size. When $\lambda=0$, the score is just $RSS$, so it does not charge for extra predictors. As $\lambda$ increases, an additional predictor must deliver a larger reduction in $RSS$ to justify its cost. A larger penalty does not guarantee that the smallest candidate wins: the result depends on the actual fit improvements as well as the penalty.

The key comparison can be derived directly. Consider a current model $M$ and a model $M'$ that adds $q$ predictors, so $k(M')=k(M)+q$. The score difference is:

\begin{aligned} J_\lambda(M')-J_\lambda(M) &=RSS(M')+\lambda\{k(M)+q\}-\{RSS(M)+\lambda k(M)\}\\ &=RSS(M')-RSS(M)+\lambda q\\ &=-\{RSS(M)-RSS(M')\}+\lambda q. \end{aligned}

The larger model has a lower score precisely when this difference is negative. Equivalently, it is preferred when its reduction in residual sum of squares is greater than the penalty for the added predictors:

J_\lambda(M')<J_\lambda(M) \quad\Longleftrightarrow\quad RSS(M)-RSS(M')>\lambda q.

For one added predictor, compare the reduction in $RSS$ with $\lambda$. For several added predictors, compare the total reduction with $q\lambda$. Equality gives a tie under this score; the score alone then does not distinguish the models.

Worked Example: A Penalty Changes the Preferred Size

A hypothetical insurer fits OLS models to the same 40 policies, using annual claim cost measured in thousands of dollars. Each candidate includes an intercept. The candidates have the following training residual sums of squares:

Model	Number of predictors, $k$	$RSS$, in squared thousands of dollars
$M_1$	1	120
$M_2$	2	90
$M_3$	3	72

Use $J_{20}(M)=RSS(M)+20k(M)$ to select a model. Then check how the choice changes if the penalty is 10 instead.

Method. Calculate the score for each candidate using the same penalty and compare the resulting values.

With $\lambda=20$:

\begin{aligned} J_{20}(M_1)&=120+20(1)=140,\\ J_{20}(M_2)&=90+20(2)=130,\\ J_{20}(M_3)&=72+20(3)=132. \end{aligned}

The lowest score is 130, so $M_2$ is preferred. The first added predictor reduces $RSS$ by $120-90=30$, which exceeds its penalty of 20. The third model's extra predictor reduces $RSS$ by $90-72=18$, which is less than 20, so moving from $M_2$ to $M_3$ raises the score by $20-18=2$.

With $\lambda=10$:

J_{10}(M_1)=130,\qquad J_{10}(M_2)=110,\qquad J_{10}(M_3)=102.

Now $M_3$ has the lowest score. Its additional reduction of 18 exceeds the new penalty of 10. The answer changes because the penalty changed, not because the candidate fits changed.

Check. At penalty 20, the scores are 140, 130 and 132, so $M_2$ is the unique minimum. At penalty 10, they are 130, 110 and 102, so $M_3$ is the unique minimum. A common mistake is to choose $M_3$ for both penalties simply because its $RSS$ is lowest; that ignores the complexity charge.

Comparing Subsets of Different Composition

The score can compare models that are not nested as well as nested ones. A subset is not defined only by its size: two models with the same number of predictors can have different residual sums of squares because they contain different predictors. First calculate each candidate's score; do not assume that a bigger model always has the lowest score or that two equally sized models fit equally well.

Worked Example: Ranking Several Candidate Subsets

A hypothetical analyst models annual repair cost for the same 60 vehicles, in thousands of dollars, using OLS with an intercept. Every candidate is fit to the same observations. The candidates are a one-predictor model $A$ using vehicle age, two-predictor models $B$ using age and annual mileage and $C$ using mileage and a region indicator, and a three-predictor model $D$ using age, mileage and the region indicator. Treat each predictor as one unit of complexity. Their residual sums of squares are $84$, $67$, $64$ and $55$, respectively. Use $\lambda=8$.

Method. Apply $J_8(M)=RSS(M)+8k(M)$ to each subset. The response scale makes the penalty units squared thousands of dollars per predictor.

\begin{aligned} J_8(A)&=84+8(1)=92,\\ J_8(B)&=67+8(2)=83,\\ J_8(C)&=64+8(2)=80,\\ J_8(D)&=55+8(3)=79. \end{aligned}

Model $D$ is preferred, with score 79. Among the two-predictor models, $C$ beats $B$ because $64<67$ and both pay the same penalty of 16. Moving from $C$ to $D$ reduces $RSS$ by $64-55=9$; the additional predictor costs 8, so the score improves by $9-8=1$.

Check. The four scores are 92, 83, 80 and 79. Model $D$ is the unique minimum. A common mistake is to compare $D$ only with $B$ because both include age and mileage. The task is to rank all stated candidates under the same rule; the best-scoring candidate here is $D$, but the calculation does not establish that it is best among models that were never considered.

The Penalty Must Match the Response Scale

Because $RSS$ is a sum of squared residuals, changing the units of the response changes the numerical size of the score's fit term. If the response is multiplied by $c$, residuals are multiplied by $c$, and $RSS$ is multiplied by $c^2$. To preserve the same model ranking under this rescaling, multiply $\lambda$ by $c^2$ as well. Keeping the penalty number fixed while changing response units changes the tradeoff.

Worked Example: Rescaling the Response Without Changing the Choice

For the same hypothetical 50 observations, two OLS candidates with intercepts have these results when the response is annual claim cost in thousands of dollars: model $E$ has $k=1$ and $RSS=80$; model $F$ has $k=2$ and $RSS=58$. Use $\lambda=12$. Then express the response in dollars and adjust the penalty so that the comparison is unchanged.

Method. Calculate both scores on the stated scale, then account for the squared change in response units.

\begin{aligned} J_{12}(E)&=80+12(1)=92,\\ J_{12}(F)&=58+12(2)=82. \end{aligned}

Model $F$ is preferred on the thousands-of-dollars scale. Converting the response to dollars multiplies every residual by $1{,}000$, so it multiplies every $RSS$ by $1{,}000^2=1{,}000{,}000$. The matching penalty is therefore $12(1{,}000{,}000)=12{,}000{,}000$. The scores in dollar units are:

\begin{aligned} J_{12{,}000{,}000}(E) &=80{,}000{,}000+12{,}000{,}000(1)=92{,}000{,}000,\\ J_{12{,}000{,}000}(F) &=58{,}000{,}000+12{,}000{,}000(2)=82{,}000{,}000. \end{aligned}

Model $F$ remains preferred. Both scores have been multiplied by $1{,}000{,}000$, so their ordering is unchanged.

Check. In the original units the scores are 92 and 82; in dollars they are 92,000,000 and 82,000,000. The choice is $F$ on either consistent scale. A common mistake is to convert $RSS$ to dollar units but leave $\lambda=12$. That would make the penalty negligible relative to the rescaled fit term and would not represent the same comparison.

What the Score Can and Cannot Establish

A generic penalized score makes an explicit compromise: a candidate earns credit for lower residual error and pays for its counted complexity. It provides a transparent ranking only for the candidates, fit procedure, sample and penalty actually specified. A winning score does not prove that the selected model is the true data-generating model, nor does it quantify the uncertainty in the selection.

The penalty also encodes a choice. Different values of $\lambda$ can select different subsets, as the first worked example showed. The score's result should therefore be reported together with the penalty and complexity definition, rather than presented as an inevitable consequence of the data. In particular, count parameters consistently: if one categorical predictor requires several coefficients, treating it as a single unit while counting other predictors by coefficient can misstate relative complexity.

Use a common basis for comparison. Keep the response, observations and fitting procedure fixed across candidates.
Check the units. The penalty and fit term must be commensurate; rescaling the response requires rescaling the penalty by the square of the unit conversion.
Use the score's direction correctly. The preferred candidate minimizes $J_\lambda$, not maximizes it.
Check the actual increment. For $q$ added predictors, compare the reduction in $RSS$ with $q\lambda$; do not decide from predictor count alone.

Key takeaway: A subset-selection decision should balance fit against complexity. Under the generic score $J_\lambda(M)=RSS(M)+\lambda k(M)$, added predictors are worthwhile only when their reduction in residual sum of squares exceeds their total penalty, with all candidates and units defined consistently.

Check Your Understanding

For each problem, assume that the candidates use the same response observations and OLS fitting procedure, and that every model includes an intercept not counted in $k$. Select the model or models with the lowest stated score.

A one-predictor model has $RSS=44$, a two-predictor model has $RSS=35$, and a three-predictor model has $RSS=32$. With $\lambda=6$, calculate all three scores and identify the preferred size. (Answer: Scores 50, 47 and 50; the two-predictor model is preferred.)
Two candidate subsets have $(k,RSS)=(1,70)$ and $(2,55)$, while a third has $(3,43)$. With $\lambda=10$, calculate the scores and decide whether the fit gain from the third predictor justifies its penalty relative to the two-predictor model. (Answer: Scores 80, 75 and 73; the three-predictor model is preferred, since the RSS reduction of 12 exceeds the penalty of 10.)
Candidate models have $(k,RSS)=(1,100)$, $(2,86)$ and $(3,73)$. Find the scores at $\lambda=14$, and state whether the score uniquely selects a model. (Answer: Scores 114, 114 and 115; the one- and two-predictor models tie for the minimum.)
On a response scale measured in thousands of dollars, a one-predictor model has $RSS=36$ and a two-predictor model has $RSS=25$. With $\lambda=8$, which model is preferred? If the response is instead measured in dollars, what penalty preserves that comparison? (Answer: The two-predictor model is preferred, with scores 44 and 41; use $\lambda=8{,}000{,}000$ in dollar units.)
A two-predictor model has $RSS=62$, and a four-predictor model has $RSS=43$. At $\lambda=7$, calculate the two scores and compare the improvement in fit with the cost of both added predictors. (Answer: The scores are 76 and 71; the four-predictor model is preferred because the RSS reduction of 19 exceeds the total penalty $2(7)=14$.)

Model	Number of predictors, \(k\)	\(RSS\), in squared thousands of dollars
\(M_1\)	1	120
\(M_2\)	2	90
\(M_3\)	3	72

What You'll Learn