Why Adding a Predictor Cannot Worsen Training Fit
In subset selection, candidate models are often compared by their training residual sum of squares (RSS). A basic fact makes those comparisons easier to interpret: if a least-squares model is expanded by adding a predictor, its minimum training RSS cannot increase. The reason is not that the new predictor must be useful. Rather, the expanded model can always reproduce every fitted value that the smaller model could produce.
This conclusion concerns training RSS under ordinary least squares. The same observations and response values must be used in both models, and the expanded model must retain all terms from the smaller model. No normality assumption is required for this algebraic result. It does not say that adding a predictor will improve prediction on new observations.
A model space is the set of all fitted-value vectors the model can produce as its coefficients vary. For example, an intercept-only model can produce any constant fitted-value vector. Adding a predictor allows more possible fitted-value vectors, but does not remove the constant ones. The least-squares fit in the expanded model therefore minimizes RSS over a set that includes all the smaller model’s available fits.
The Nested-Model Result
Let \(Z\) be the design matrix for a smaller model, including an intercept if the model uses one. Let \(X\) be the design matrix for a larger model that retains every column of \(Z\) and adds at least one predictor. Both models use the same response vector \(y\) and the same observations, and both are fit by unweighted ordinary least squares.
To justify the inequality, take any coefficient vector \(\beta\) for the smaller model. In the expanded model, assign those same coefficients to the columns from \(Z\) and assign zero to every added predictor. The fitted values are then exactly \(Z\beta\), so the expanded model can attain the smaller model’s RSS. Since least squares chooses the best fit available in the expanded model, its minimum cannot be larger.
For any coefficients \(\beta\), its fitted-value vector is \(Z\beta\).
Use the same coefficients on the retained columns and set every added coefficient to zero. The expanded model then also fits \(Z\beta\).
The expanded model minimizes over a set containing every smaller-model fit, so its minimum RSS is less than or equal to the smaller model’s minimum RSS.
The argument does not require the design matrix to have full column rank. If columns are redundant, some coefficient vectors may not be unique, but the set of possible fitted values—and hence the minimum RSS—remains well-defined. The key requirement is that the larger model really does retain the smaller model’s terms, with the same response and observations.
How Much Can One Added Predictor Reduce RSS?
The set-inclusion argument establishes the direction of change. A projection argument gives the amount of change when one predictor is added. Let \(P_Z\) be the orthogonal projection matrix onto the column space of \(Z\), and let \(M_Z=I-P_Z\). The smaller model’s residual vector is \(r=M_Zy\). For the added predictor column \(x\), define \(z=M_Zx\), the part of that predictor not already explained by the smaller model.
If \(z\) is nonzero, the expanded model adds the direction \(z\) to the smaller model’s space. Since \(r\) is orthogonal to that smaller space, fitting along the added direction amounts to choosing a scalar \(\alpha\) to minimize \(\|r-\alpha z\|^2\). Expanding the squared norm gives
This is a quadratic in \(\alpha\). Its minimum occurs at \(\hat{\alpha}=z'r/(z'z)\). Substitution gives the expanded-model RSS and its reduction:
Because \(z'z>0\) and \((z'r)^2\geq 0\), the reduction is nonnegative. It is zero exactly when \(z'r=0\): after removing the part already explained by the smaller model, the added predictor is orthogonal to the smaller model’s residuals. If \(z=0\), the added predictor is already in the smaller model’s column space; it introduces no new fitted-value direction, and RSS is unchanged.
Worked Calculations
Worked Example: Adding a Predictor to an Intercept-Only Model
For three hypothetical policy records, the responses are \(y=(2,4,4)\). Compare an intercept-only model with a model that also includes predictor values \(x=(-1,0,1)\). Calculate both training RSS values and verify the direction of change.
Method. Fit each model by ordinary least squares on these same three observations. The intercept-only fitted value is the response mean:
The intercept-only residuals are \(2-\frac{10}{3}=-\frac{4}{3}\), \(4-\frac{10}{3}=\frac{2}{3}\), and \(4-\frac{10}{3}=\frac{2}{3}\). Therefore,
For the model with an intercept and \(x\), the predictor is centered because \((-1)+0+1=0\). Its least-squares slope is the sum of \(x_i(y_i-\bar y)\) divided by the sum of \(x_i^2\):
The fitted intercept is \(\bar y=\frac{10}{3}\). Thus the fitted values are \(\frac{10}{3}-1=\frac{7}{3}\), \(\frac{10}{3}\), and \(\frac{10}{3}+1=\frac{13}{3}\). The residuals are \(-\frac{1}{3}\), \(\frac{2}{3}\), and \(-\frac{1}{3}\), so
The expanded model’s RSS is \(2/3\), below the intercept-only RSS of \(8/3\). The reduction is \(8/3-2/3=2\). As a check using the reduction formula, the centered predictor has squared length \(2\), and its inner product with the intercept-only residuals is \(2\), giving a reduction of \(2^2/2=2\).
Common mistake avoided: The numerator in the slope calculation is \(2\), not the slope itself. Divide by the predictor’s sum of squares, also \(2\), to obtain slope \(1\).
Worked Example: Recomputing Fits After Adding a Second Predictor
For four hypothetical records, let the response be \(y=(4,4,3,2)\), with first predictor \(x_1=(2,0,-2,0)\) and added predictor \(x_2=(0,0,2,0)\). Compare the least-squares model with an intercept and \(x_1\) to the model with an intercept, \(x_1\), and \(x_2\).
For the smaller model, the response mean is \(\bar y=13/4\). Since the entries of \(x_1\) sum to zero, its fitted slope is \(x_1'y/(x_1'x_1)\):
The smaller-model fitted values are \(13/4+(1/4)x_1=(15/4,13/4,11/4,13/4)\). Subtracting these from \(y\) gives residuals \((1/4,3/4,1/4,-5/4)\). Thus
In the expanded model, coefficients \(\hat{\beta}_0=3\), \(\hat{\beta}_1=1/2\), and \(\hat{\beta}_2=1/2\) give fitted values \(3+\frac12x_1+\frac12x_2\). Evaluate each row: \(3+1+0=4\); \(3+0+0=3\); \(3-1+1=3\); and \(3+0+0=3\). In particular, the third fitted value is \(3\).
The resulting residuals are \((0,1,0,-1)\). They sum to zero, their inner product with \(x_1\) is \(0\), and their inner product with \(x_2\) is \(0\). These are the least-squares orthogonality conditions for the intercept and both predictor columns, so these fitted values are indeed the expanded model’s least-squares fit. Its RSS is \(0^2+1^2+0^2+(-1)^2=2\), which is less than \(9/4\).
Common mistake avoided: Evaluate the fitted-value formula using each row’s actual predictor values before calculating residuals. A single incorrect fitted value changes the residual and RSS calculations that follow.
Worked Example: An Added Predictor Can Leave RSS Unchanged
For four hypothetical observations, let \(y=(3,5,2,6)\), and compare an intercept-only model with a model that adds \(x=(-1,-1,1,1)\). Determine whether training RSS decreases.
The mean response is \((3+5+2+6)/4=4\). The intercept-only residuals are \((-1,1,-2,2)\), so the smaller-model RSS is \(1+1+4+4=10\). The added predictor sums to zero, and its inner product with those residuals is
Also, \(x'x=1+1+1+1=4\), so the least-squares slope is \(0/4=0\). The expanded-model fitted values remain \((4,4,4,4)\), and the expanded RSS remains \(10\). The added predictor provides a new direction, but that direction has no alignment with the current residuals.
Common mistake avoided: “Cannot increase” does not mean “must decrease.” A predictor can have zero estimated slope and produce exactly the same training RSS.
What This Result Does—and Does Not—Say
The guarantee applies to the optimized training RSS when the larger model contains the smaller model. It may fail as a comparison if the models use different observations, different weights, or different fitting criteria. It also does not apply if a term is removed while another is added and neither model contains the other: then their model spaces need not be nested.
Most importantly, lower training RSS alone is not evidence that an expanded model predicts better on new data. The extra flexibility can fit random features of the training sample. Model assessment therefore needs criteria beyond raw training RSS; the monotonicity result explains why training RSS alone will not penalize model size.
- Check that both models use the same response observations and least-squares objective.
- Check that every term in the smaller model is retained in the expanded model.
- Interpret an unchanged RSS as possible, especially when the new predictor adds no residual-aligned direction.
- Do not infer lower test error solely from lower training RSS.
This property is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus because it links model nesting, least-squares geometry, and the interpretation of a training-fit criterion.
Check Your Understanding
For each problem, compare least-squares training RSS values or apply the one-predictor reduction formula.
- For \(y=(1,3,5)\), compare an intercept-only model with a model adding \(x=(-1,0,1)\). Find both RSS values. (Answer: intercept-only RSS \(=8\); expanded-model RSS \(=0\).)
- For \(y=(2,4,4)\), compare an intercept-only model with a model adding \(x=(-1,0,1)\). Find both RSS values. (Answer: intercept-only RSS \(=8/3\); expanded-model RSS \(=2/3\).)
- For \(y=(3,5,2,6)\), an intercept-only model has residuals \((-1,1,-2,2)\). If the added predictor is \(x=(-1,-1,1,1)\), calculate its slope and the expanded RSS. (Answer: slope \(=0\); expanded RSS \(=10\).)
- A current model has residual vector \(r=(1,-1,0,0)\). The part of a proposed predictor not explained by the current model is \(z=(1,1,-1,-1)\). The current RSS is \(2\). Find the expanded RSS. (Answer: \(2\); the reduction is \(0\).)
- A current model has residual vector \(r=(2,0,-1,-1)\) and RSS \(6\). The residualized added predictor is \(z=(1,-1,1,-1)\). Find the RSS reduction and expanded RSS. (Answer: reduction \(=1\); expanded RSS \(=5\).)