Why Compare Nested Regression Models?
A regression model may impose restrictions on a more general model. For example, a risk analyst might ask whether a group of coefficients can all be set to specified values without materially worsening the model’s fit. Comparing the models’ residual sums of squares (SSEs) provides a way to measure the loss of fit caused by those restrictions. The partial F test scales that loss by both the number of restrictions and the residual variation left in the less restricted model.
The test is exact under the normal linear model assumptions stated below. Its result is not simply a comparison of two SSEs: the numerator must account for how many independent restrictions are imposed, and the denominator must use the residual degrees of freedom from the full model. This derivation is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
Ranks count linearly independent columns in the design matrix, including an intercept column when one is present. In a full-rank model, \(q\) is also the number of independent coefficient restrictions being tested. The models must use the same response observations and the same response scale for their SSEs to be comparable.
Model, Hypothesis, and Test Statistic
Assume the full model is \(Y=X_F\beta+\varepsilon\), where \(Y\) is the \(n\)-vector of responses, \(X_F\) is a fixed design matrix of rank \(r_F\), and the errors are independent and normally distributed with mean zero and common variance \(\sigma^2\). Equivalently, \(\varepsilon\sim N(0,\sigma^2 I)\). The restricted model is nested in the full model, and the null hypothesis states that the mean response satisfies the restrictions defining the restricted model.
Under the null hypothesis, the true mean response is available within the restricted model. The full model can fit at least as well as the restricted model because it allows all the restricted model’s fitted values, and possibly more. Thus \(SSE_R-SSE_F\) is nonnegative. The partial F statistic is
The numerator is the reduction in SSE per independent restriction. The denominator is the full model’s residual mean square, an estimate of \(\sigma^2\) based on the full model’s residual degrees of freedom. Under the null hypothesis and the stated assumptions, this statistic has an \(F\) distribution with \(q\) numerator and \(n-r_F\) denominator degrees of freedom.
Why the Partial F Statistic Has an F Distribution
Let \(P_R\) and \(P_F\) be the projection matrices onto the column spaces of the restricted and full design matrices. A projection matrix maps a response vector to its least-squares fitted values. Because the models are nested, the restricted column space is contained in the full column space, so \(P_F-P_R\) is a symmetric idempotent matrix of rank \(q=r_F-r_R\). The residual sums of squares can be written as \(SSE_R=Y'(I-P_R)Y\) and \(SSE_F=Y'(I-P_F)Y\). Subtracting gives
Under the null hypothesis, the mean vector \(\mu=E[Y]\) belongs to the restricted column space. Therefore, \((P_F-P_R)\mu=0\). The difference above is then a quadratic form in the normal error vector. A symmetric idempotent matrix of rank \(q\), applied to a normal vector whose mean is annihilated by that matrix, gives a chi-square quadratic form with \(q\) degrees of freedom when divided by \(\sigma^2\):
The full-model residual matrix \(I-P_F\) is also symmetric and idempotent, with rank \(n-r_F\). Since the full model contains the restricted space, the ranges of \(P_F-P_R\) and \(I-P_F\) are orthogonal. The corresponding normal quadratic forms are independent. Consequently,
The ratio is an F statistic because it is the ratio of two independent chi-square variables after each has been divided by its degrees of freedom; the common factor \(\sigma^2\) cancels. Normality is needed for this exact finite-sample result. If the errors are not normal, the statistic may still be used as an approximation in suitable settings, but its exact F distribution is not guaranteed.
Calculating and Interpreting the Statistic
A reliable calculation keeps the rank difference and residual degrees of freedom separate. First find \(q=r_F-r_R\). Then find \(\nu=n-r_F\), not \(n-r_R\). Next calculate the mean reduction in SSE, \((SSE_R-SSE_F)/q\), and the full-model residual mean square, \(SSE_F/\nu\). Their ratio is the test statistic. A large value indicates that the restrictions cause a large increase in residual error relative to the variation remaining in the full model.
Worked Example: Testing Three Restrictions in a Risk Model
Hypothetical scenario. An analyst models quarterly claim-cost variation using the same 32 observations for two nested linear models. The restricted model has rank \(r_R=4\) and \(SSE_R=150\); the full model has rank \(r_F=7\) and \(SSE_F=90\). At the 5% significance level, use the supplied critical value \(F_{3,25}=2.99\) to test the restrictions.
Method. Use the partial F statistic. The number of independent restrictions is \(q=r_F-r_R=7-4=3\). The full model’s residual degrees of freedom are \(\nu=n-r_F=32-7=25\).
The reduction in SSE is \(150-90=60\), so the reduction per restriction is \(60/3=20\). The full-model residual mean square is \(90/25=3.6\). Therefore,
Since \(5.5556>2.99\), reject the null hypothesis at the 5% level. The data provide evidence that the three restrictions, considered jointly, reduce fit too much to be supported by this test. This conclusion does not identify which individual restriction is responsible.
Check. The full model has three more independent columns, so \(q=3\); its residual degrees of freedom are \(32-7=25\). Also, the nested full model’s SSE, 90, is below the restricted model’s SSE, 150. The calculation avoids treating the total reduction of 60 as if it were already a mean square.
Worked Example: Using the Correct Residual Degrees of Freedom
Hypothetical scenario. A pricing analyst compares two nested models fitted to 28 policy-period observations. The restricted model has rank 5 and SSE 84. The full model has rank 7 and SSE 63. A supplied 5% critical value for \(F_{2,21}\) is 3.47. Calculate the test statistic and make the decision.
Method. There are \(q=7-5=2\) independent restrictions. The denominator degrees of freedom come from the full model, so \(\nu=28-7=21\). The SSE reduction is \(84-63=21\), giving a numerator mean square of \(21/2=10.5\). The full-model residual mean square is \(63/21=3\).
Because \(3.5>3.47\), reject the null hypothesis at the 5% level. The restrictions are jointly inconsistent with the observed fit at this significance level.
Check. The statistic uses \(F_{2,21}\): two restrictions and \(28-7=21\) full-model residual degrees of freedom. The value 3 is the full-model residual mean square, calculated from \(63/21\).
Worked Example: Avoiding the Restricted-Model Denominator
Hypothetical scenario. Two nested models for a portfolio’s monthly loss experience use the same 36 observations. The restricted model has rank 3 and SSE 144; the full model has rank 6 and SSE 72. Calculate the partial F statistic and identify the error in using the restricted model’s residual mean square as the denominator.
Method. The restriction count is \(q=6-3=3\), and the full-model residual degrees of freedom are \(\nu=36-6=30\). The SSE reduction per restriction is \((144-72)/3=72/3=24\). The correct denominator is the full-model residual mean square, \(72/30=2.4\). Thus,
Common mistake. Using the restricted model’s residual degrees of freedom gives \(36-3=33\), and its residual mean square is \(144/33=4.3636\). Dividing 24 by this value gives approximately \(5.5\), which is not the partial F statistic. The correction is to use \(SSE_F/(n-r_F)=72/30=2.4\), because the denominator estimates the error variance from the full model whose residual variation remains after the restrictions are relaxed.
Check. The nested full model’s SSE is lower, and the numerator uses three restrictions. Using the prescribed full-model denominator gives \(24/2.4=10\).
What the Test Does—and Does Not—Establish
The partial F test evaluates a joint null hypothesis: whether all the restrictions hold together. Rejecting it says that the collection of restrictions is not supported as a whole under the model assumptions. It does not show that every restriction is false, nor does it determine which one matters most. Failing to reject is not proof that the restrictions are true; it means the data do not provide sufficient evidence against them at the chosen significance level.
The comparison requires genuinely nested models, the same response and observations, and compatible least-squares fits. If models are not nested, the difference \(SSE_R-SSE_F\) does not have the chi-square numerator derivation given above. Likewise, if the models use different observations, differences in SSE can reflect different data as well as different restrictions. Check nesting and the observation set before calculating.
Check Your Understanding
For each problem, identify the restriction count and full-model residual degrees of freedom before calculating or interpreting the statistic.
- Two nested models use \(n=24\) observations and have ranks 4 and 6. Their SSEs are 96 for the restricted model and 60 for the full model. Find the partial F statistic. (Answer: \(q=2\), \(\nu=18\), \(F=5.4000\).)
- A nested-model comparison has \(n=50\), restricted rank 5, full rank 8, \(SSE_R=210\), and \(SSE_F=126\). Find the statistic. (Answer: \(q=3\), \(\nu=42\), \(F=9.3333\).)
- A comparison has four independent restrictions, 16 full-model residual degrees of freedom, \(SSE_R=100\), and \(SSE_F=60\). Calculate the statistic. (Answer: \(F=(40/4)/(60/16)=2.6667\).)
- Two models use the same response but are not nested. Is \((SSE_R-SSE_F)/q\), divided by the full model’s residual mean square, justified as a partial F statistic by the derivation here? (Answer: No; the nesting condition needed for the projection-matrix derivation is absent.)
- For \(n=40\), a restricted model of rank 4 has SSE 180, and a nested full model of rank 7 has SSE 108. Calculate \(q\), the full-model residual degrees of freedom, and \(F\). (Answer: \(q=3\), \(\nu=33\), \(F=7.3333\).)