Why the Analysis Sample Matters
Subset selection compares candidate regression models using a score. That comparison is meaningful only when the scores refer to sufficiently comparable data. Missing values can undermine that condition: a model using predictors \(X_1\) and \(X_2\) may be fitted to different observations from a model using \(X_1\) and \(X_3\). A score difference can then reflect not only the predictors, but also which cases happened to be included.
This issue is easy to miss when software automatically drops any row with a missing value among the variables in a particular model. Such deletion is often called complete-case fitting. It can produce a different complete-case sample for each candidate model. The resulting regressions may each be valid least-squares fits for their own observations, but their scores do not automatically make a fair head-to-head comparison.
For ordinary least squares, the residual sum of squares for model \(M\) on its analysis sample is
Here \(y_i\) is the observed response and \(\hat{y}_{i,M}\) is the fitted value for observation \(i\) from model \(M\). If two models have different analysis samples, their RSS values sum squared errors over different sets of response values. The smaller number is not, by itself, evidence that one model predicts better.
Why a Shared Sample Makes the Comparison More Interpretable
Suppose two models are fitted by least squares to exactly the same response observations. If one model is nested in the other, the larger model can reproduce every fitted-value pattern available to the smaller model, and may have additional possibilities. This gives a precise reason that RSS cannot increase when a predictor is added on the same sample.
when \(M_1\) is nested in \(M_2\), both use the same set \(I\), and \(\mathcal{B}_1\subseteq\mathcal{B}_2\) describes their available fitted-value patterns. The inequality follows because minimizing over a larger set cannot produce a larger minimum.
That argument depends on the common set \(I\). When the observations differ, the two models minimize different sums, so the nested-model argument does not compare their RSS values. Even when models are not nested, using the same observations removes this particular source of unfairness. It does not establish that a model is scientifically appropriate or that its advantage will persist in new data.
Worked Example: A Smaller RSS From Fewer Observations
A hypothetical claims team is considering two regressions for a response measured in thousands of dollars. Model A includes a complete building-age variable and has \(n_A=8\). Model B includes a safety-audit score, which is missing on some records, and has \(n_B=4\). The fitted residuals for A on its eight observations are \(3,-3,3,-3,3,-3,3,-3\). The fitted residuals for B on its four observations are \(1,-1,2,-2\). Compare the RSS values and state what can be concluded.
Method. Calculate each model's RSS on its own analysis sample, then identify the limitation of comparing those totals. The common mistake is to treat the smaller total as proof that B fits better.
For Model A, each of the eight residuals has square \(9\), so
For Model B, the squared residuals are \(1,1,4,4\), giving
The raw RSS is smaller for B, but B was assessed on only four observations rather than eight. The totals do not establish that B is superior. Dividing by the sample sizes gives \(72/8=9\) and \(10/4=2.5\), respectively, but these are still averages over different observations. That division does not make the samples comparable.
A fairer comparison is to identify one common set of eligible observations, fit both models on that set, and compare their scores there. The calculations above are checked: \(72\) is the sum of eight squared residuals of \(9\), and \(10\) is \(1+1+4+4\). No model ranking follows from those two numbers alone.
Different Observations Can Reverse a Ranking
The problem is not limited to comparing models with different sample sizes. Two samples of equal size may contain different observations, with different response values and relationships between predictors and response. In that case, equal sample sizes do not make the RSS values comparable either.
A practical approach for a subset-selection comparison is to define the candidate predictor universe first, then determine which observations have complete values for the response and all predictors in that universe. Fit each candidate on that common set. This controls the observation set while the predictor subset changes. It may reduce the available sample substantially, so report the resulting sample size and the predictors whose missingness caused the loss of cases.
Worked Example: Equal Sample Sizes, Different Cases
Two candidate regressions for a hypothetical monthly benefit amount each have eight complete observations, but the records are not the same. Model A has RSS \(20\); Model B has RSS \(25\). The five observations shared by both candidate samples are then identified. After refitting each model on those same five observations, the residual-square contributions are reported as \(3,3,4,4,4\) for A and \(1,1,2,2,4\) for B. Which model has lower RSS on the common sample?
Method. First recognize that the initial totals cover different records. Then compare the refitted models using the five shared observations. The common mistake is to infer from \(20<25\) that A must have the smaller RSS on the cases used for comparison.
The original comparison has A lower by \(25-20=5\), but it is not a fair comparison because the eight observations differ. On the common sample, Model A has
Model B has
Thus B has the lower common-sample RSS by \(18-10=8\). The ranking differs from the apparent ranking of the original totals. The common-sample arithmetic is checked directly: A's five contributions sum to \(18\), while B's sum to \(10\). The result illustrates why the original \(20\)-versus-\(25\) comparison was not decisive.
R-Squared Also Depends on the Sample
For a model with an intercept, the usual in-sample coefficient of determination is \(R^2=1-\operatorname{RSS}/\operatorname{SST}\), where \(\operatorname{SST}=\sum_{i\in I}(y_i-\bar{y}_I)^2\) is the total sum of squares around the mean response in that sample. Because both RSS and SST are computed from the analysis sample, \(R^2\) values from different samples need not compare the same response variation.
The next example uses summary values from hypothetical regression output. Each model's native-sample \(R^2\) is calculated correctly, but the comparison changes after both models are refitted on shared observations.
Worked Example: R-Squared Ranking Changes on a Common Sample
For two candidate regressions, Model A's own analysis sample has \(\operatorname{SST}=100\) and \(\operatorname{RSS}=40\). Model B's different analysis sample has \(\operatorname{SST}=100\) and \(\operatorname{RSS}=50\). On the common observations, the total sum of squares is \(80\) for both models; after refitting, A has RSS \(36\) and B has RSS \(32\). Calculate both native-sample \(R^2\) values and both common-sample \(R^2\) values.
Method. Apply \(R^2=1-\operatorname{RSS}/\operatorname{SST}\) separately to each sample. The common mistake is to treat values based on different observations as if they were calculated against the same response variation.
On their own samples, Model A has
Model B has
Those values might suggest that A has the better fit, but the models used different records. On the common sample, A has
B has
On the common observations, B has the larger \(R^2\), \(0.60\) versus \(0.55\). The calculations are checked by dividing each RSS by its stated SST before subtracting from one. The native-sample ranking is not a fair basis for choosing between the models; the common-sample values provide the controlled comparison in this example.
How to Set Up a Common-Sample Comparison
Use a consistent procedure before comparing candidate subset models. A common sample should be based on the full set of predictors that might appear in the candidates, not separately on each model's selected predictors. Otherwise, changing the subset can silently change the observations being scored.
Record the variables that may be used in the comparisons and ensure that every candidate uses the same response definition.
Keep observations with observed response values and observed values for every predictor in the candidate universe. Record the resulting sample size.
Estimate each candidate regression using the same response observations. Do not compare an earlier fit from a candidate-specific complete-case sample with these refitted models.
Use the same observations and the same score definition for each candidate. State the comparison's sample size and remember that the result applies to that analysis sample.
Limitations and Common Exam Traps
A common sample addresses one issue: whether competing models are scored on the same observations. It does not guarantee that the retained observations represent the target population. If cases with missing values differ systematically from retained cases, a complete-case analysis can still be unrepresentative. Nor does the common-sample rule recover information in missing values. The suitability of a missing-data method requires assumptions and analysis beyond a simple score comparison.
Keep these distinctions clear when interpreting regression output:
- Smaller RSS with fewer observations: The total is often smaller simply because fewer squared residuals are added. It is not a direct measure of comparative model quality across different samples.
- Same \(n\), different cases: Equal sample sizes do not mean equal response values or equal predictor-response relationships. The observation identities must match, not just the counts.
- R-squared from different samples: Each sample has its own response mean and total sum of squares. A higher native-sample \(R^2\) does not ensure a higher \(R^2\) on shared observations.
- Common sample means no missing-data bias: It does not. A common complete-case sample makes the model comparison controlled with respect to observations, but may change which population the analysis represents.
Check Your Understanding
For each problem, distinguish what can be concluded from candidate-specific scores and what requires a common-sample comparison.
- Model C has RSS \(48\) on 12 records; Model D has RSS \(30\) on 6 different complete records. Can the raw RSS values establish which model fits better? (Answer: No; the models use different observations and sample sizes.)
- Two candidate models each use 20 observations, but only 15 observations are shared. Is equal sample size sufficient for a fair RSS comparison? (Answer: No; their observation sets differ.)
- On a common sample with \(\operatorname{SST}=120\), a refitted model has RSS \(42\). Calculate its \(R^2\). (Answer: \(1-42/120=0.65\).)
- On separate samples, Model E has \(\operatorname{SST}=80\), RSS \(24\), and Model F has \(\operatorname{SST}=100\), RSS \(35\). Calculate each native-sample \(R^2\), and state whether the comparison is controlled. (Answer: E: \(0.70\); F: \(0.65\); the comparison is not controlled because the samples differ.)
- A common-sample analysis discards 30% of records because one candidate predictor is frequently missing. Does this make all resulting comparisons invalid, or does it settle whether the retained sample represents the target population? (Answer: Neither conclusion follows automatically; comparisons use the same retained cases, but representativeness still requires consideration.)