Why Better In-Sample Fit Does Not Guarantee Usefulness

Why Fit Is Not the Same as Usefulness

Regression fit summarizes how closely a model’s fitted values match the observations used to estimate it. That information matters, but it answers a limited question: how well did this model describe these data? It does not, by itself, tell us whether the model will predict new outcomes well, whether its predictors contain useful information, or whether its coefficients support the interpretation someone wants to make.

This distinction is especially important when comparing models on the same outcome and sample. A more flexible model can often reduce its residual sum of squares (RSS), and, when the outcome varies, increase its R-squared. Those improvements are properties of fitting the observed sample. Additional evidence and sound reasoning are needed to judge broader usefulness.

What the Fit Measures Say

Definition: For observations \(y_1,\ldots,y_n\) and fitted values \(\hat{y}_1,\ldots,\hat{y}_n\), the residual sum of squares is \(RSS=\sum_{i=1}^{n}(y_i-\hat{y}_i)^2\). It measures squared discrepancies between observed and fitted outcomes, in squared outcome units. In a regression model with an intercept, when \(TSS=\sum_{i=1}^{n}(y_i-\bar{y})^2>0\), R-squared is \(R^2=1-RSS/TSS\). If \(TSS=0\), meaning all observed outcomes are equal, this definition of R-squared is undefined.

For models fitted to the same observed outcomes, \(TSS\) is fixed. Therefore, a smaller RSS corresponds to a larger R-squared when \(TSS>0\). Neither measure accounts by itself for the number of predictors, how the model performs on new data, or whether its terms have a defensible interpretation. RSS also depends on the scale of the outcome, so its size is meaningful in context and is not directly comparable across outcomes measured on different scales.

In this discussion, a model is a working description of the conditional mean of an outcome given its predictors, such as \(E(Y_i\mid X_i)=\beta_0+\beta_1X_{i1}+\cdots+\beta_pX_{ip}\). Least squares chooses coefficient estimates to minimize RSS in the sample. That fitting operation does not establish that the mean form is correct, that a predictor has a real association beyond the sample, or that the association is causal. Such claims require appropriate design, assumptions, and evidence.

Why Adding Predictors Can Improve Training Fit

For ordinary least squares, consider a smaller model and a larger model that includes every predictor in the smaller model plus one or more additional terms. Both models must use the same outcome and the same observations for the comparison to have this property. The larger model can always reproduce the smaller model’s fitted values by setting the added coefficients to zero. Since least squares selects the coefficients that minimize RSS among the choices available, the larger model’s minimum RSS cannot be greater.

RSS_{\text{larger}}\leq RSS_{\text{smaller}}

This is a statement about optimization on the fitting sample, not proof that the added terms help outside it. The same logic applies to R-squared when \(TSS>0\): for these nested models on the same outcome and sample, R-squared cannot decrease. The improvement can be substantial when a useful pattern is captured, but it can also be a response to chance irregularities in the sample.

Justification: Every coefficient choice available to the smaller model is available to the larger one: use the smaller model’s coefficients and set all added coefficients to zero. Thus, the minimum RSS over the larger set of choices is no greater than the minimum over the smaller set. If \(TSS>0\) is the same for both models, \(R^2=1-RSS/TSS\) gives the corresponding non-decrease in R-squared.

Worked Example: A Flexible Curve Fits Better In Sample

Suppose a hypothetical training sample has four observations: \((x,y)=(0,0),(1,1),(2,2),(3,20)\). Compare the least-squares straight line with a cubic curve that passes through all four points. This small example illustrates training fit, not a general rule about which model should be chosen.

For the straight line, \(\bar{x}=1.5\), \(\bar{y}=5.75\), \(S_{xx}=\sum(x_i-\bar{x})^2=5\), and \(S_{xy}=\sum(x_i-\bar{x})(y_i-\bar{y})=30.5\). Its slope is \(30.5/5=6.1\), and its intercept is \(5.75-6.1(1.5)=-3.4\). Thus, \(\hat{y}=-3.4+6.1x\). Its fitted values at the four observed \(x\)-values are \(-3.4,2.7,8.8,14.9\), giving residuals \(3.4,-1.7,-6.8,5.1\). Therefore,

RSS_{\text{line}}=3.4^2+(-1.7)^2+(-6.8)^2+5.1^2 =11.56+2.89+46.24+26.01=86.70.

A cubic curve that passes through these points is \(\hat{y}=x+\frac{17}{6}x(x-1)(x-2)\). At \(x=0,1,2\), the product term is zero and the fitted values equal \(0,1,2\); at \(x=3\), it adds \(\frac{17}{6}(3)(2)(1)=17\) to 3, giving 20. The cubic has zero residuals and \(RSS_{\text{cubic}}=0\), a lower training RSS than the line.

That exact fit does not show that the cubic is more useful. At \(x=4\), the line predicts \(-3.4+6.1(4)=21\), while the cubic predicts \(4+\frac{17}{6}(4)(3)(2)=72\). If a separate hypothetical observation at \(x=4\) had outcome 4, the absolute prediction errors would be \(|21-4|=17\) and \(|72-4|=68\), respectively. In this illustration the more flexible model fits the training observations exactly but predicts the separate observation much less well. A training fit cannot substitute for checking performance on data not used to fit the model.

Incidental Terms and Unsupported Meaning

A predictor can lower RSS even when its observed association is accidental or unhelpful for the intended purpose. For example, a randomly generated identifier might happen to align with part of the residual pattern in one sample. Its added term can improve training fit without representing a stable relationship. That improvement alone is not evidence that the identifier will help predict future outcomes, and it certainly does not give the identifier a meaningful scientific interpretation.

Worked Example: A Small RSS Reduction From a Meaningless Predictor

In a hypothetical data set of 40 observations, suppose an intercept-only model has \(TSS=200\) and \(RSS=120\). A larger model adds a randomly generated code that has no substantive meaning; in this sample its RSS is 111.2. Because both models use the same outcome and observations, we can calculate the fit summaries directly.

For the intercept-only model, \(R^2=1-120/200=0.400\). For the model with the code, \(R^2=1-111.2/200=0.444\). The RSS reduction is \(120-111.2=8.8\), or \(8.8/120\approx0.0733\), a 7.33% reduction relative to the initial RSS. The larger model therefore fits these observations better according to both RSS and R-squared.

The calculations do not show that the code is a useful predictor. The fit improvement could reflect a sample-specific coincidence, and the code has no sensible role in explaining the outcome. To assess predictive usefulness, one would need evidence about performance on observations not used to fit the model and a reason to expect the relationship to persist. A smaller training RSS alone provides neither.

A further limitation is that a useful description of association is not automatically a causal explanation. Even when a predictor improves fit and its coefficient has a clear numerical meaning, other variables or the process that assigned predictor values may account for the observed relationship. A causal conclusion requires a design and assumptions that support it; a good in-sample fit is not a substitute.

Worked Example: An Association Does Not Establish a Cause

Suppose a hypothetical sample of five hospitals records staffing hours per patient per day (\(x\)) and readmission percentages (\(y\)). The pairs are \((20,15),(22,14),(24,13),(26,13),(28,10)\). A straight-line regression gives an association between staffing and readmission in this sample.

Here \(\bar{x}=24\) and \(\bar{y}=13\). The deviations in \(x\) are \(-4,-2,0,2,4\), and those in \(y\) are \(2,1,0,0,-3\). Thus, \(S_{xy}=(-4)(2)+(-2)(1)+(0)(0)+(2)(0)+(4)(-3)=-22\), while \(S_{xx}=(-4)^2+(-2)^2+0^2+2^2+4^2=40\). The fitted slope is \(-22/40=-0.55\), and the intercept is \(13-(-0.55)(24)=26.2\). The fitted line is \(\hat{y}=26.2-0.55x\).

The estimated slope describes a sample association: the fitted readmission percentage is 0.55 percentage points lower per additional staffing hour per patient per day. It does not show that increasing staffing would cause readmissions to fall. Hospital size, patient case mix, or other differences might be related to both staffing and readmission. The fitted relationship is not, on its own, a causal argument.

How to Judge Whether Better Fit Is Useful

A fit statistic is one piece of model assessment, not a final verdict. When a more complex model has lower RSS on the fitting data, ask what the comparison does and does not establish. Useful follow-up questions include:

Will the model generalize? Assess predictions on suitable observations not used to estimate the model, rather than relying only on its training fit.
Do the added terms have a defensible role? Consider whether they represent information that would be available and meaningful for the intended use.
Does the interpretation match the evidence? Describe an estimated association as an association unless the study design and assumptions justify a causal claim.
Is the comparison fair? The non-increase result for RSS applies to nested least-squares models fitted to the same outcome and observations. It does not by itself compare models fitted to different samples or outcomes.

There is no universal RSS decrease that makes a model useful. The practical question depends on what the model is meant to do and whether its performance and interpretation hold up beyond the fitting calculation. The key is to separate the numerical fact that fit improved from the broader claim that the model is valuable.

Common Mistake

Common mistake: “The model has a larger R-squared, so its added predictors are useful and the model is better.”

Correction: For nested least-squares models on the same outcome and sample, adding predictors cannot increase RSS and cannot decrease R-squared when \(TSS>0\). That mechanical fit improvement does not establish better prediction on new observations, meaningful predictors, or a valid substantive or causal interpretation.

Check Your Understanding

Use the definitions and distinctions in this tutorial to answer the following questions.

Why can a nested least-squares model with additional predictors not have a larger minimum RSS when fitted to the same observations?
A model’s RSS falls when a new predictor is added. What can you conclude about its training fit, and what remains unresolved about its usefulness?
In the cubic example, why does an exact fit to the four training observations fail to establish that the cubic will predict well at \(x=4\)?
When is the stated definition \(R^2=1-RSS/TSS\) undefined, and what feature of the observed outcomes causes this?
A regression slope links staffing hours to readmission percentage. What additional issue must be addressed before describing the association as a causal effect?