Why Compare Two Regression Models?
A regression analysis often begins with a smaller model that includes predictors already considered important. The question may then be whether adding another predictor—or a group of predictors—improves the model enough to support including those terms. Software can fit both models, but its output is useful only if the comparison is set up and interpreted correctly.
A nested-model comparison tests a specific set of added terms while retaining the terms in a reduced model. Its partial F test assesses whether those additions explain enough additional variation in the response, relative to the remaining unexplained variation, to be inconsistent with the null hypothesis. It does not by itself show that a relationship is causal or that the larger model is preferable for every purpose.
What to Identify in Software Output
Before reading a comparison table, identify the response, the predictors in each model, and which terms are new in the full model. For example, if the reduced model includes age and treatment group, while the full model also includes baseline score and an age-by-treatment interaction, the additions are baseline score and that interaction. The null hypothesis concerns their coefficients together, not each coefficient in isolation.
For an ordinary least-squares regression with an intercept, let \(n\) be the number of observations and let \(k_R\) and \(k_F\) be the numbers of slope coefficients in the reduced and full models. If the full model adds \(q=k_F-k_R\) coefficients, its residual degrees of freedom are \(df_F=n-k_F-1\), provided the full model’s design matrix (including the intercept) has full column rank and the added terms raise the rank by \(q\). In general, the residual degrees of freedom equal \(n\) minus the rank of the full design matrix, and \(q\) is the increase in rank. The partial F statistic can be calculated from the two residual sums of squares:
Here, \(SSE_R\) and \(SSE_F\) are the residual sums of squares for the reduced and full models. The numerator is the decrease in residual sum of squares per added coefficient; the denominator is the full model’s residual mean square. Under the null hypothesis and the assumptions below, this statistic follows an \(F\) distribution with \(q\) numerator degrees of freedom and \(df_F\) denominator degrees of freedom. A software comparison table may report this as an “F change,” “partial F,” or “model comparison” test.
How the Partial F Test Is Justified
The full model cannot have a larger SSE than the reduced model when both are ordinary least-squares fits to the same observations and the full model contains all the reduced model’s terms: the reduced fit remains available by setting the added coefficients to zero. Thus \(SSE_R-SSE_F\) measures the improvement in fit from allowing the additions. The test asks whether that improvement is large relative to the full model’s residual variation.
Under the null hypothesis that all \(q\) added coefficients are zero, and under the stated normal-error assumptions, the improvement divided by the error variance has a chi-square distribution with \(q\) degrees of freedom. The full-model residual sum of squares divided by the error variance has a chi-square distribution with \(df_F\) degrees of freedom, and these quantities are independent. Therefore, dividing each by its degrees of freedom and taking their ratio gives the F reference distribution:
The p-value is the upper-tail probability \(P(F_{q,df_F}\geq F_{\mathrm{obs}})\), where \(F_{\mathrm{obs}}\) is the calculated statistic. A small p-value is evidence against the joint null hypothesis: the data would be unusual if the added coefficients were all zero and the model assumptions held. It is not the probability that the null hypothesis is true.
Worked Example: Reading a Two-Predictor Addition
In hypothetical data on 33 patients, a team models a numeric outcome using a reduced model with age and treatment group, then fits a full model that also includes baseline score and an age-by-treatment interaction. Both fits use all 33 patients. The comparison output is:
| Model | Residual df | Residual sum of squares |
|---|---|---|
| Reduced: age, treatment group | 30 | 500 |
| Full: reduced terms, baseline score, age-by-treatment interaction | 28 | 390 |
Added terms and hypotheses. The full model adds two coefficients, so \(q=2\). The null hypothesis is that the baseline-score coefficient and the age-by-treatment interaction coefficient are both zero, conditional on age and treatment group. The alternative is that at least one of these two coefficients is nonzero.
Assumptions and degrees of freedom. The output states that both models use the same outcome and the same 33 observations, and the terms make the reduced model a special case of the full model. The full model has 28 residual degrees of freedom, consistent with an intercept and four slope coefficients fitted to 33 observations. The usual regression error assumptions are needed for the exact F test.
Calculation. The full model reduces SSE by \(500-390=110\). Therefore,
The reference distribution is \(F(2,28)\). Its upper-tail p-value is approximately \(0.0309\). At significance level \(\alpha=0.05\), the result is statistically significant. In context, the data provide evidence that adding baseline score and the age-by-treatment interaction, considered jointly, improves the model for the outcome beyond age and treatment group alone. The test does not determine which added term is responsible for the evidence; that is a separate question.
Reading the Test in Context
A careful report connects the model comparison to the study question. State what terms were added, the joint null hypothesis, the test statistic and its degrees of freedom, the p-value, and the conclusion at the chosen significance level. For example, “After accounting for the predictors in the reduced model, the added terms were jointly associated with a statistically significant improvement in fit” is more informative than “the full model was significant.”
The phrase “after accounting for” reflects the conditional nature of multiple regression. It does not mean that the added variables were randomly assigned or that their coefficients describe causal effects. The partial F test also does not measure how large or practically important the improvement is. SSE reduction and measures such as \(R^2\) can describe aspects of fit, but statistical significance alone is not a measure of practical value.
Worked Example: A Non-Significant Block of Additions
Suppose hypothetical data on 48 clinics are used to predict an outcome. The reduced model contains two predictors, and the full model adds two more predictors. The models use the same clinics, and software reports \(SSE_R=310\), \(SSE_F=280\), and \(df_F=43\). Assume the regression conditions for the F test are reasonable.
The full model adds \(q=2\) coefficients. The null hypothesis is that both added coefficients are zero conditional on the two reduced-model predictors. Calculate:
The reference distribution is \(F(2,43)\), giving an upper-tail p-value of approximately \(0.112\). At \(\alpha=0.05\), this is not statistically significant. The data do not provide sufficient evidence that the two added predictors jointly improve the model, after accounting for the reduced-model predictors. This conclusion is not proof that both coefficients are exactly zero; it is a statement about the evidence supplied by these data and this test.
Related Output: One Added Coefficient
When the full model adds exactly one coefficient, \(q=1\), its partial F test is equivalent to the two-sided t test of that single coefficient, provided both tests use the same model and error estimate. The statistics satisfy \(F=t^2\). This is a useful check when software provides both a coefficient table and a model-comparison table. With multiple additions, the partial F test is a joint test and cannot generally be replaced by inspecting separate t tests one at a time.
Worked Example: Checking a One-Predictor Comparison
For hypothetical data on 60 patients, a reduced model contains two predictors. A full model adds one predictor, and both models use the same outcome and observations. Software reports \(SSE_R=900\), \(SSE_F=810\), and \(df_F=56\). Assume the usual regression conditions for the exact F test.
Here \(q=1\). The null hypothesis is that the added predictor’s coefficient is zero, conditional on the two predictors already in the model. The comparison gives:
With \(F(1,56)\), the upper-tail p-value is approximately \(0.01559\). At \(\alpha=0.05\), there is evidence that the added predictor improves the model conditional on the existing predictors. Since \(F=t^2\), the corresponding coefficient test has \(|t|=\sqrt{6.222}\approx2.494\) and the same two-sided p-value. The F statistic has no sign, so the direction of the estimated relationship must be read from the coefficient estimate, not from this comparison.
Common Mistakes and a Practical Reading Sequence
A frequent mistake is to treat any pair of regression outputs as comparable. If the models use different observations—for example, because one predictor has missing values—the difference in SSE does not isolate the contribution of the added terms. Likewise, an F comparison of nested models is not the appropriate test for two models where neither can be obtained from the other by setting coefficients to zero.
Another mistake is to interpret a small p-value as proof that every added predictor matters. The joint test only assesses whether all added coefficients are zero together. If a categorical predictor is added using several indicator coefficients, for instance, the block test concerns those coefficients jointly; an individual coefficient test asks a different question.
Use this sequence when reading a software comparison:
Record the response and list the predictors in the reduced and full models. Identify exactly which terms are added.
Verify that the models are nested and use the same response, observations, and fitting method.
Find the number \(q\) of added coefficients, the full-model residual degrees of freedom, and either the reported F statistic or the two residual sums of squares needed to calculate it.
Use the upper-tail p-value, compare it with the stated \(\alpha\), and give a conclusion about the added terms in context without claiming causation or practical importance solely from significance.
Check Your Understanding
Use the definitions and model-comparison steps above to answer the following.
- A full model contains all the terms in a reduced model plus three coefficients. What is \(q\), and what joint null hypothesis does the partial F test assess?
- Why must the reduced and full models be fitted to the same observations for the usual SSE-based comparison?
- A comparison reports \(F=2.7\) with \(q=2\) and \(df_F=35\). Which tail of the F distribution gives the p-value?
- What does a statistically significant joint test establish, and what does it not establish about the individual added predictors?
- When one coefficient is added, how are its partial F statistic and its coefficient t statistic related?