Why Compare Nested Models?
A regression model with more predictors can fit the observed data at least as closely as a model with fewer predictors. That improvement alone does not show that the added terms are useful: adding parameters generally reduces the residual sum of squares, even when the apparent improvement is mostly random variation. A nested-model ANOVA comparison asks whether the reduction is large relative to the remaining residual variation.
For example, a hypothetical insurer might model claim severity using policy characteristics, then consider adding regional indicators or interaction terms. The statistical question is whether the added terms jointly improve the model, not whether each term is individually significant. This comparison is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
The Partial F Test for Nested Models
Suppose the full model adds \(q\) coefficients to the reduced model. The null hypothesis says those added coefficients are all zero; the alternative says that at least one is nonzero. More generally, the null can specify other linear restrictions that make the reduced model a special case of the full one. The test compares the average reduction in residual sum of squares per added parameter with the full model’s residual mean square.
Here \(n\) is the number of observations, \(q\) is the numerator degrees of freedom, and \(n-p_F\) is the full model’s residual degrees of freedom. A large \(F\) value indicates that the reduction in residual sum of squares is large relative to the full model’s residual variation. The p-value is the upper-tail probability \(P(F_{q,n-p_F}\ge F_{\mathrm{observed}})\). Reject \(H_0\) when the p-value is at most the chosen significance level.
Why the Formula Works
The test relies on the usual linear-model assumptions for exact finite-sample inference: the models are correctly specified and nested, have full column rank, use the same observations, and errors are independent and normally distributed with mean zero and common variance \(\sigma^2\). The variance is unknown and estimated from the full model. If these assumptions do not hold, the stated exact F reference distribution need not apply.
The logic follows from how the two sums of squares behave under the null. The smaller model cannot fit better than the larger one by least squares on the same data, so \(\mathrm{RSS}_R-\mathrm{RSS}_F\ge 0\). Under \(H_0\), the reduction measures the variation attributable to the \(q\) restrictions that the full model relaxes. With normal errors, that reduction divided by \(\sigma^2\) has a chi-square distribution with \(q\) degrees of freedom. The full-model residual sum of squares divided by \(\sigma^2\) has a chi-square distribution with \(n-p_F\) degrees of freedom; these two quantities are independent.
Under \(H_0\), \((\mathrm{RSS}_R-\mathrm{RSS}_F)/\sigma^2\) has a chi-square distribution with \(q\) degrees of freedom. Dividing by \(q\) gives an estimate of the error variance based on the improvement.
\(\mathrm{RSS}_F/\sigma^2\) has a chi-square distribution with \(n-p_F\) degrees of freedom. Dividing \(\mathrm{RSS}_F\) by \(n-p_F\) gives the full model’s residual mean square.
The two scaled chi-square quantities are independent under the assumptions. Their ratio after division by their respective degrees of freedom therefore has an \(F_{q,n-p_F}\) distribution under \(H_0\).
This justification explains two details that are easy to misread: the numerator degrees of freedom count the restrictions being tested, while the denominator uses the full model’s residual degrees of freedom. The reduction is not divided by the reduced model’s residual degrees of freedom.
Worked Comparisons
Worked Example: Testing Two Added Risk Factors
Hypothetical scenario. A claim-severity study uses \(n=24\) observations. The reduced regression has an intercept and one predictor, so \(p_R=2\). The full regression adds two risk factors, so \(p_F=4\). Their residual sums of squares are \(\mathrm{RSS}_R=180\) and \(\mathrm{RSS}_F=120\). Test at the 5% level whether the two added factors jointly improve the model. Assume the errors are independent \(N(0,\sigma^2)\) with common unknown variance; interest and mortality assumptions are not applicable.
Method. Use the partial F test. The models are nested and use the same 24 observations. The number of added parameters is \(q=4-2=2\), and the full-model residual degrees of freedom are \(24-4=20\).
The reduction in residual sum of squares is \(180-120=60\). Its mean square is \(60/2=30\). The full-model residual mean square is \(120/20=6\). Thus
The reference distribution under \(H_0\) is \(F_{2,20}\). For this distribution the upper-tail probability at 5.00 is \((20/(20+2(5)))^{10}=(2/3)^{10}\approx0.0173\). Since \(0.0173<0.05\), reject \(H_0\). The added factors provide statistically significant improvement at the 5% level, though this alone does not establish that the improvement is practically important.
Common mistake avoided: Do not use \(p_F-p_R\) as the denominator degrees of freedom. It is the numerator degrees of freedom; the denominator degrees of freedom are \(20\).
Worked Example: A Small Reduction Is Not Automatically Significant
Hypothetical scenario. A model for a portfolio’s monthly claim count uses \(n=30\) observations. A reduced model has \(p_R=3\) coefficients, and a nested full model has \(p_F=5\). The respective residual sums of squares are 150 and 138. Test at the 5% level. Assume independent normal errors \(N(0,\sigma^2)\) with common unknown variance; interest and mortality assumptions are not applicable.
Method. Compare the average reduction for the two added coefficients with the full model’s residual mean square. Here \(q=5-3=2\), and the full-model residual degrees of freedom are \(30-5=25\).
The reduction is \(150-138=12\), giving an improvement mean square of \(12/2=6\). The full-model residual mean square is \(138/25=5.52\). Therefore
The reference distribution is \(F_{2,25}\). Its upper-tail probability at this value is \((25/(25+2(1.087)))^{12.5}\approx0.353\). Since \(0.353>0.05\), do not reject \(H_0\). The data do not provide sufficient evidence at the 5% level that the two added coefficients jointly improve the model.
Common mistake avoided: A decrease in RSS from 150 to 138 is not, on its own, evidence of a significant improvement. The reduction must be judged against the full model’s residual mean square and the appropriate degrees of freedom.
Worked Example: Testing Added Curvature and an Interaction
Hypothetical scenario. A pricing analyst fits a regression to \(n=18\) observations. The reduced model has an intercept and two main effects, so \(p_R=3\). The full model also includes a squared term and an interaction, so \(p_F=5\). The reduced and full residual sums of squares are 84 and 60. Test at the 5% level. Assume independent errors with distribution \(N(0,\sigma^2)\), where the common variance is unknown; interest and mortality assumptions are not applicable.
Method. The full model contains the reduced model’s terms, so this is a joint test of the two additional coefficients. The numerator degrees of freedom are \(q=5-3=2\). The full-model residual degrees of freedom are \(18-5=13\).
The improvement is \(84-60=24\), and its mean square is \(24/2=12\). The full-model residual mean square is \(60/13\approx4.6154\). The test statistic is
For \(F_{2,13}\), the p-value is \((13/(13+2(2.60)))^{13/2}=(13/18.2)^{6.5}\approx0.112\). Since \(0.112>0.05\), do not reject the joint null hypothesis at the 5% level. The lower RSS does not establish that the curvature and interaction terms jointly improve the model enough to exceed the residual variation.
Common mistake avoided: Do not perform two separate one-coefficient tests if the question concerns whether the two terms jointly add value. The joint partial F test uses \(q=2\) and directly tests both restrictions together.
Interpretation and Common Traps
A nested-model F test is a comparison of fit after accounting for model size. Adding coefficients cannot increase the least-squares RSS when the models use the same response observations and the smaller model is nested within the larger one. Consequently, the raw difference in RSS is nonnegative, but its size must be assessed relative to the full model’s residual mean square. A large reduction can be unconvincing when residual variation is also large; a smaller reduction can be persuasive when residual variation is small.
| Quantity | Correct choice | Reason |
|---|---|---|
| Improvement sum of squares | \(\mathrm{RSS}_R-\mathrm{RSS}_F\) | Measures the fit gained by moving to the full model |
| Numerator degrees of freedom | \(q=p_F-p_R\) | Counts the added coefficients or restrictions tested |
| Residual mean square | \(\mathrm{RSS}_F/(n-p_F)\) | Uses the full model’s residual variation and degrees of freedom |
| Reference distribution | \(F_{q,n-p_F}\) | Applies under the stated linear-model assumptions |
A frequent error is to put \(\mathrm{RSS}_R/(n-p_R)\) in the denominator. For the first worked example, that would give a denominator of \(180/(24-2)=180/22\approx8.1818\), and a mistaken statistic of \(30/8.1818\approx3.667\). The correct denominator is the full-model residual mean square \(120/(24-4)=6\), giving \(F=30/6=5.00\). The test compares the improvement with the residual variation remaining after fitting the full model.
The comparison is not valid as stated if the models are not nested or if they were fitted to different observations, such as when adding a predictor introduces missing-data exclusions. Also, failure to reject does not prove that the reduced model is true; it means the data do not provide enough evidence against the specified restrictions at the chosen significance level. Conversely, statistical significance does not by itself establish causation, predictive usefulness, or material importance.
Check Your Understanding
For each problem, identify the numerator and denominator degrees of freedom before deciding what the result says.
- A reduced model has \(p_R=2\), a full nested model has \(p_F=3\), and both use \(n=20\) observations. Their RSS values are 96 and 72, respectively. Calculate the partial F statistic and test at 5%. (Answer: \(F=5.667\) with degrees of freedom \((1,17)\); \(p\approx0.029\), so reject \(H_0\) at 5%.)
- Two nested models use \(n=25\) observations, with \(p_R=3\), \(p_F=5\), \(\mathrm{RSS}_R=200\), and \(\mathrm{RSS}_F=170\). Calculate \(F\) and decide at 5%. (Answer: \(F\approx1.765\) with degrees of freedom \((2,20)\); \(p\approx0.197\), so do not reject \(H_0\).)
- A full model with \(p_F=6\) is nested above a reduced model with \(p_R=4\); both use \(n=16\) observations. If their RSS values are 112 and 70, calculate the test statistic and decision at 5%. (Answer: \(F=3.00\) with degrees of freedom \((2,10)\); \(p\approx0.095\), so do not reject \(H_0\).)
- A reduced model has an intercept and three predictors; a full model adds four terms. Both models use the same \(n=40\) observations, and their RSS values are 310 and 250. Calculate \(F\) and decide at 5%. (Answer: \(q=4\), full-model residual degrees of freedom \(32\), and \(F=1.92\); do not reject \(H_0\) at 5%.)
- Two regressions have RSS values 110 and 92, but one was fitted to 28 observations and the other to 25 because the added predictor has missing values. Can the standard nested-model partial F formula be applied directly to these reported RSS values? (Answer: No; the models must be compared on the same response observations, so refit them to a common sample before applying the standard test.)