Common Pitfalls in Reporting Regression Fit

Introduction

A regression result can be calculated correctly and still be reported misleadingly. A statement such as “the model explains 70% of the outcome” may leave out whether that percentage refers to the observed sample, which outcome was modeled, or what “explains” means. A claim that one model “fits significantly better” can also hide which models were compared and which predictors were tested.

Good reporting makes the comparison identifiable and keeps the conclusion within the evidence. For fit statistics, state the outcome, the sample, the measure, and the model. For a nested-model test, also state the reduced and expanded models, the block of added terms, and the test result. Neither a large fit statistic nor a small p-value, by itself, establishes that a model is useful for prediction or that a predictor causes a change in the outcome.

What a Fit Statistic Describes

Definition: In an ordinary least-squares regression with an intercept, $R^2$ is the fraction of the sample’s total variation in the outcome around its sample mean accounted for by the fitted regression model. If $SSE$ is the sum of squared residuals and $SST$ is the total sum of squares around the outcome mean, then $R^2=1-SSE/SST$. Adjusted $R^2$ modifies $R^2$ to account for the number of predictors fitted.

The interpretation of $R^2$ is tied to a particular outcome and sample. For example, $R^2=0.64$ means that, in the data used to fit the model, the model accounts for 64% of the observed variation in the outcome around its sample mean, according to the least-squares decomposition. It does not mean that the model predicts every individual outcome with 64% accuracy, that 64% of people are correctly classified, or that the predictors caused 64% of the variation.

Adjusted $R^2$ is a related summary that penalizes fitting additional predictors. Let $n$ be the number of observations and $k$ the number of predictors, not counting the intercept. For a model with an intercept, it is calculated as

R^2_{\text{adj}}=1-(1-R^2)\frac{n-1}{n-k-1}.

The adjustment can make adjusted $R^2$ lower than $R^2$, and it can decrease when predictors are added. It is a fit summary, not a hypothesis test. A difference between adjusted $R^2$ values does not, on its own, tell us whether a block of added coefficients is statistically significant.

Worked Example: Reporting an R-Squared Value in Context

Suppose a hypothetical regression model uses $n=80$ observations to predict weekly commute time in minutes from two measured predictors. Its $SST$ is 1,250,000 minutes-squared and its $SSE$ is 450,000 minutes-squared. The model includes an intercept.

First calculate the fraction of sample variation accounted for by the model:

R^2=1-\frac{SSE}{SST} =1-\frac{450{,}000}{1{,}250{,}000} =1-0.36 =0.64.

A precise report is: “In the 80 observations used to fit the model, the regression of weekly commute time in minutes on the two predictors had $R^2=0.640$, so the fitted model accounted for 64.0% of the sample variation in commute time around its mean.” This describes in-sample fit. It does not claim that the model will predict commute times accurately for new commuters, or that either predictor caused differences in commute time.

Comparing Adjusted R-Squared Carefully

Adjusted $R^2$ is most interpretable when the models use the same outcome measured in the same way and the same observations. If the sample changes, the total variation and the relationships in the data may change too. The resulting values then mix changes in the model with changes in the data being summarized.

When the outcome and sample match, a higher adjusted $R^2$ indicates a higher adjusted in-sample fit summary among the models compared. It does not establish that the improvement matters for a particular decision, that predictions will generalize, or that the expanded model is preferable under every goal. Report both values and identify the candidate models rather than announcing that one is “best” without qualification.

The adjustment follows directly from the residual degrees of freedom. With an intercept, the residual mean square is $SSE/(n-k-1)$, while the total mean square around the outcome mean is $SST/(n-1)$. The adjusted measure can be written as one minus the ratio of these two quantities:

R^2_{\text{adj}} =1-\frac{SSE/(n-k-1)}{SST/(n-1)} =1-(1-R^2)\frac{n-1}{n-k-1}.

Thus, adding predictors changes not only $SSE$, but also the residual degrees of freedom. This penalty is why adjusted $R^2$ can fall even though ordinary $R^2$ cannot fall when predictors are added to a least-squares model fit to the same data.

Worked Example: An Adjusted R-Squared Increase Is Not a Test

Consider two hypothetical models for the same outcome, using the same $n=60$ observations and an intercept. The reduced model has $k=2$ predictors and $R^2=0.480$. The expanded model has $k=5$ predictors and $R^2=0.510$.

For the reduced model:

R^2_{\text{adj}} =1-(1-0.480)\frac{60-1}{60-2-1} =1-(0.520)\frac{59}{57} \approx 0.462.

For the expanded model:

R^2_{\text{adj}} =1-(1-0.510)\frac{60-1}{60-5-1} =1-(0.490)\frac{59}{54} \approx 0.465.

The expanded model has the higher adjusted $R^2$, approximately 0.465 rather than 0.462, and the ordinary $R^2$ also rises from 0.480 to 0.510. A defensible report gives both values and the shared sample and outcome. The small adjusted-$R^2$ increase is not itself a p-value, does not prove that the added block is useful, and does not guarantee better performance on new observations. A nested-model test addresses a different question.

Reporting a Nested-Model Test

A nested-model test compares a reduced model with an expanded model that contains every term in the reduced model plus a specified block of additional terms. The test asks whether the population coefficients for that block are all zero, conditional on the terms retained in the reduced model. Both models must use the same outcome and observations for the usual partial F-test comparison to be interpretable.

Definition: Suppose the expanded model adds $q$ coefficients to the reduced model. The partial F statistic is $$ F=\frac{(SSE_R-SSE_F)/q}{SSE_F/(n-k_F-1)}, $$ where $SSE_R$ and $SSE_F$ are the residual sums of squares for the reduced and expanded models, $k_F$ is the number of predictors in the expanded model excluding the intercept, and $n-k_F-1$ is its residual degrees of freedom. Under the null hypothesis that all $q$ added coefficients are zero, this statistic has an $F$ distribution with $q$ and $n-k_F-1$ degrees of freedom, under the linear-model conditions for the test.

For the usual exact test, the regression model must be correctly specified as linear in its coefficients, errors must be independent with mean zero and constant variance, and the errors must be normally distributed conditional on the predictors. The hypotheses concern the entire added block: $H_0$ states that all its coefficients are zero, and $H_a$ states that at least one is nonzero. The test’s p-value is the probability, assuming $H_0$ and the model conditions, of obtaining an $F$ statistic at least as large as the observed one.

The numerator is the reduction in residual sum of squares per added restriction. The denominator estimates the error variance from the expanded model. Their ratio therefore measures whether the reduction in residual variation is large relative to the residual variation remaining in that model. A small p-value is evidence against the joint null hypothesis; it is not a measure of how large or practically useful the improvement is.

Worked Example: Reporting a Partial F-Test Without Overclaiming

In a hypothetical sample of 50 observations, a reduced model contains three predictors and an intercept. An expanded model adds two predictors, so $q=2$ and $k_F=5$. Both models use the same outcome and all 50 observations. Suppose $SSE_R=240$ and $SSE_F=210$, in squared units of the outcome.

The expanded model has $50-5-1=44$ residual degrees of freedom. The observed partial F statistic is

F=\frac{(240-210)/2}{210/44} =\frac{15}{4.7727} \approx 3.143.

For an $F$ distribution with 2 and 44 degrees of freedom, the upper-tail p-value is approximately 0.053. At significance level $\alpha=0.05$, this result does not meet the criterion for rejecting the joint null hypothesis. A suitable report is: “For the 50 observations analyzed, the two added predictors were compared with the reduced model using a partial F-test; $F(2,44)=3.143$, $p\approx0.053$. At the 0.05 level, the data do not provide sufficient evidence that the added coefficients are jointly different from zero.” This is not proof that both coefficients are zero or that the expanded model has no value.

The test also does not say that the added predictors improve the model by a large amount. For context, one could report the residual sums of squares and the comparison’s partial $R^2$, calculated here as $(240-210)/240=0.125$. That is a sample fit summary for the added block relative to the reduced model; it does not replace the test or settle whether the change is useful for a particular purpose.

Common Reporting Pitfalls

A concise regression report should make clear what was fit and what the evidence supports. The following errors often turn a correct calculation into an inaccurate conclusion.

“The model explains 64% of the outcome.” This wording can suggest that 64% of individual outcomes are explained or that the predictors caused the outcome. Say instead that the model accounted for 64% of the sample variation in the specified outcome around its mean.
“Adjusted $R^2$ increased, so the new predictors are significant.” Adjusted $R^2$ is not a hypothesis test. If the question is whether a specified block of coefficients is jointly zero, report the nested-model test, including its degrees of freedom and p-value.
“The expanded model fits significantly better.” Without naming the models, sample, added terms, statistic, and p-value, the reader cannot tell what was tested. Also avoid implying that statistical evidence alone establishes practical usefulness.
Comparing models fitted to different observations. If one model omits cases with missing predictor values, its $R^2$ may use a different sample from the other model. Refit both models on the same cases for a direct nested comparison, and state the sample size.
Treating a large $R^2$ as proof of a good prediction model. In-sample fit does not by itself establish performance on new data. State that limitation rather than presenting $R^2$ as a measure of out-of-sample accuracy.

A practical reporting checklist is: name the outcome and its units; state the sample size and which observations were used; identify the model or model comparison; report the relevant fit statistic or test result; and describe the conclusion narrowly. For a nested test, specify the added block and its degrees of freedom. Mention relevant limitations, including that the fit summary is in-sample and that an observational regression result does not by itself establish causation.

Key takeaway: Report regression fit as a property of a named model fitted to a stated outcome and sample. Report a nested test as evidence about a specified block of coefficients. Neither a fit statistic nor a p-value alone establishes predictive usefulness, practical importance, or causation.

Check Your Understanding

Use the definitions and examples above to answer these questions.

A model has $R^2=0.72$ for a sample of 100 observations. What does this value describe, and name one conclusion it does not justify.
Two nested models have different adjusted $R^2$ values. Why is the difference not itself a test of whether the added coefficients are jointly zero?
What must match between two models for a usual nested-model comparison to be interpretable, and what else should be stated in a report?
A partial F-test gives $p=0.03$ at $\alpha=0.05$. What evidence does this provide, and what does it not establish about the size or usefulness of the fit improvement?
Why can comparing $R^2$ values from models fitted to different samples be misleading?