Why Compare Models at the Same Size?
A subset-selection search can produce several candidate regression models with the same number of predictors. Each candidate has the same size, but its predictors differ. The direct question at this stage is narrow: which candidate fits the observed response best among models of this size? Answering that question does not, by itself, determine how many predictors the final model should contain.
For a fair fit comparison, the candidates must be fitted to the same response values on the same observations. In this tutorial, each candidate is an ordinary least-squares linear regression with an intercept. The number of predictors, denoted by \(k\), counts the explanatory variables and does not count the intercept. No probability distribution for the errors is required to compare the observed training fit. If a Gaussian regression model is used for inference, its usual assumption is independent errors with mean zero and common variance, often written \(\varepsilon_i\sim N(0,\sigma^2)\); that distributional assumption is not needed for the calculations below.
Residual Sum of Squares as the Common Fit Score
Let \(y_i\) be the observed response for observation \(i\), and let \(\hat{y}_i\) be its fitted value from a candidate model. Its residual is \(e_i=y_i-\hat{y}_i\), and its residual sum of squares is the sum of squared residuals over the \(n\) observations:
For two candidates fitted to the same response observations, the smaller RSS means that the fitted values are closer to the observed responses in squared-error terms. Thus, at a fixed predictor count, rank candidates from smallest to largest RSS. This is a training-fit comparison: the RSS is calculated on the observations used to fit the models.
The comparison is only meaningful if the values being summed refer to the same response cases. Even if two models each contain \(k\) predictors, their RSS values need not be comparable when one is fitted to a different set of observations. Predictor count alone does not make the analysis samples identical.
Worked Example: Finding the Best Two-Predictor Candidate
A hypothetical pricing study fits three ordinary least-squares models, each with an intercept and \(k=2\) predictors, to the same 60 policies and the same response, annual claim cost. The reported RSS values are 84, 96, and 89 for candidates A, B, and C, respectively. Which candidate has the best training fit at this size? The values are stipulated model summaries; no probability distribution is assumed.
Method. Because the response cases and predictor count are common, compare the RSS values directly:
Candidate A has the smallest RSS, so A is the best-fitting candidate among these three at \(k=2\). The differences from A are \(89-84=5\) for C and \(96-84=12\) for B.
Check. The direction is correct: smaller squared residual total means closer in-sample fits. The comparison does not show that two predictors are preferable to one or three; it only identifies the winner among the candidates shown at size two. The common mistake to avoid is treating “best at \(k=2\)” as “best final model.”
Training R-Squared Gives the Same Ranking Under a Condition
For a response observed on a fixed sample, define its total sum of squares by
where \(\bar{y}\) is the sample mean of the response. When \(\operatorname{TSS}>0\), the usual training coefficient of determination is
For candidates fitted to the same response values, \(\operatorname{TSS}\) is a common, positive constant. The equation therefore gives a direct reason that minimizing RSS and maximizing training \(R^2\) produce the same ranking:
The values \(y_i\), their mean \(\bar{y}\), and hence TSS are the same for every candidate.
When \(\operatorname{TSS}>0\), dividing RSS by TSS preserves its ordering: a smaller RSS gives a smaller \(\operatorname{RSS}/\operatorname{TSS}\).
A smaller ratio gives a larger \(1-\operatorname{RSS}/\operatorname{TSS}\), so the candidate with the smallest RSS has the largest training \(R^2\).
The positive-TSS condition matters. If the response is constant across the sample, then every \(y_i=\bar{y}\), so \(\operatorname{TSS}=0\). The usual formula for \(R^2\) then divides by zero and is undefined. One must not claim that maximizing this \(R^2\) ranks the candidates in that case.
Worked Example: Ranking One-Predictor Models With R-Squared
A hypothetical analyst compares three one-predictor ordinary least-squares models, each with an intercept, using the same 48 records of a numeric outcome. The common TSS is 500. The candidates’ RSS values are 310, 275, and 290. Find the training \(R^2\) for each and identify the best candidate at \(k=1\). These are observed fit summaries; no distributional model is assumed.
Step 1: Check whether the conversion applies. Since \(\operatorname{TSS}=500>0\), calculate \(R^2=1-\operatorname{RSS}/500\) for each candidate.
Step 2: Calculate candidate A.
Step 3: Calculate candidate B.
Step 4: Calculate candidate C.
Candidate B has the largest training \(R^2\), \(0.45\), and also the smallest RSS, 275. Therefore B is the best of these candidates at size one. Check: all three values use the same denominator, 500, and the ranking agrees with the RSS ranking. The common mistake is to treat a larger \(R^2\) as evidence that the model is best across all predictor counts; this calculation only compares the candidates shown at \(k=1\).
Same Predictor Count Is Not the Same as Same Final Model
Subset selection separates two questions. First, for a specified size \(k\), which subset gives the best fit among the candidates considered? Second, which model size should be retained for the analysis? The first is a within-size comparison. It can be answered by the common fit score above. The second requires a decision across model sizes and cannot be answered merely by identifying the best candidate at one size.
For example, a search may report a best one-predictor subset, a best two-predictor subset, and a best three-predictor subset. The word “best” in each result is conditional on the stated size. Choosing one of those models as final requires a separate basis for weighing fit against model size or the intended predictive and explanatory purpose. A within-size RSS ranking does not supply that basis.
Also distinguish predictor count from the identity of the predictors. Two models can both have \(k=2\) while using entirely different variables. The common score ranks their fits; it does not establish that one predictor is causally important, that the selected variables are stable under changes to the data, or that the observed difference is statistically significant.
Worked Example: Correcting a Comparison With Different Samples
Two hypothetical models each contain an intercept and two predictors. Model A was initially fitted to 40 records and has RSS \(72\). Model B was fitted to 38 records and has RSS \(68\). Two records have missing values for a predictor in B. The analyst claims B is better because \(68<72\). On the shared 38 records, refitting A gives RSS \(70\), refitting B gives RSS \(68\), and the common TSS is \(160\). Which comparison is valid, and what does it show? Treat these as stipulated OLS summaries; no error distribution is assumed.
Step 1: Identify the invalid comparison. The original RSS values use different samples: \(72\) is based on 40 records and \(68\) on 38. Their raw difference, \(72-68=4\), does not establish which model has the smaller training error on a common set of responses.
Step 2: Use the common sample and refitted models. The comparable values on the same 38 records are \(70\) for A and \(68\) for B. Thus \(68<70\), so B has the smaller RSS on the shared sample.
Step 3: Verify using training \(R^2\). Because the common TSS is positive, calculate:
B again ranks higher on the common sample: \(0.575>0.5625\). Check. The RSS and \(R^2\) rankings agree because both candidates use the same 38 outcomes and the same positive TSS. The correction is not to compare the original totals more carefully; it is to fit and score the candidates on a common analysis sample. The initial wrong calculation, \(68<72\), mixes sample sizes and cannot support the claimed ranking.
When R-Squared Cannot Be Used
If all observed responses have the same value, then \(\operatorname{TSS}=0\). Under the usual formula, \(R^2=1-\operatorname{RSS}/\operatorname{TSS}\) is undefined. This is not a case in which \(R^2\) equals zero or one by convention for ordinary model comparison. RSS can still be calculated and reported, but the usual \(R^2\)-based ranking is unavailable.
Worked Example: Constant Response and Undefined R-Squared
A hypothetical data set contains 12 observations, and every observed response is \(7\). Two candidate models each include an intercept and one predictor. The ordinary least-squares fit of each model has RSS \(0\), because the fitted intercept-only value of \(7\) reproduces every response exactly. Determine TSS and whether training \(R^2\) can rank the models. No probability distribution is being imposed.
Step 1: Calculate the response mean. Since all 12 responses are 7, \(\bar{y}=7\).
Step 2: Calculate TSS.
Step 3: Check the \(R^2\) formula. Substituting the stated RSS gives \(1-0/0\), which is undefined. Therefore, \(R^2\) cannot rank these candidates. Their RSS values tie at zero, so RSS also supplies no preference between them in this example.
Check. The obstruction is the zero denominator, not a computational rounding issue. The common mistake is to say that both models have \(R^2=1\) because their RSS is zero. That conclusion is invalid when TSS is also zero.
Practical Checklist
Before reporting a fixed-size winner, verify the conditions behind the comparison and state its scope clearly.
- Confirm that each candidate has the same predictor count \(k\), counting predictors but not the intercept.
- Confirm that every RSS was computed from the same response observations.
- Use the smallest RSS as the best training-fit score at that fixed size.
- If using training \(R^2\), confirm that the response sample is common and \(\operatorname{TSS}>0\); then the largest \(R^2\) has the same ranking as the smallest RSS.
- Describe the result as “best among the candidates at size \(k\),” rather than as the final model unless a separate model-size decision has been made.
These checks keep a fit comparison precise. The conclusion is about observed squared-error fit among equal-size candidates on a common sample. It is not, on its own, a conclusion about model size, predictive performance on new observations, causation, or statistical significance. This distinction is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
Check Your Understanding
For each problem, state the comparison being made and check the conditions before interpreting the score.
- Three models each contain an intercept and four predictors and are fitted to the same 75 responses. Their RSS values are 142, 137, and 145. Identify the best candidate at this size and state what this does not determine. (Answer: The model with RSS 137 is best among these candidates at \(k=4\); this does not select the final model size.)
- Two equal-size models use the same response sample, with \(\operatorname{TSS}=320\) and RSS values 176 and 192. Compute both training \(R^2\) values and identify the better training fit. (Answer: \(R^2=0.4500\) for RSS 176 and \(R^2=0.4000\) for RSS 192; the first model fits better.)
- Two equal-size models have RSS values 51 and 48, but they were fitted to different sets of observations. Can the lower RSS alone establish the winner? Give the required correction. (Answer: No; refit and compare both models on the same response observations.)
- On a common sample, TSS is 90 and two equal-size models have RSS values 36 and 27. Calculate the \(R^2\) values and give the ranking. (Answer: The \(R^2\) values are \(0.6000\) and \(0.7000\), respectively; the model with RSS 27 ranks higher.)
- A response is constant across the sample, and two equal-size intercept models each have RSS zero. Can the usual \(R^2\) compare them, and what does RSS say? (Answer: No; TSS is zero, so the usual \(R^2\) is undefined. Their RSS values tie, so RSS gives no preference.)