Deriving the Partial F Test for Added Model Terms

Why Compare Nested Regression Models?

A regression model can be expanded by adding one or more terms, such as additional predictors or functions of predictors. The expanded model will fit the observed data at least as well as the smaller model in the sense that its residual sum of squares cannot increase. The key question is whether the reduction is large relative to the residual variation that remains. The partial F test answers this question for a specified block of added terms.

“Partial” emphasizes that the block is tested after accounting for the terms already in the reduced model. It is a joint test: evidence is assessed for the added terms as a group, not one coefficient at a time. The comparison is meaningful only when the two models are nested and fitted to the same response values.

Models, Parameters, and the Null Hypothesis

Let \(y\) be the vector of \(n\) observed responses. Write the full model as \(y=X_F\beta+\varepsilon\), where \(X_F\) is its design matrix, \(\beta\) is its coefficient vector, and \(\varepsilon\) is the error vector. The reduced model is nested in the full model if every fitted mean vector allowed by the reduced model is also allowed by the full model. Equivalently, the column space of the reduced design matrix \(X_R\) is contained in the column space of \(X_F\).

Let \(p_R\) and \(p_F\) be the ranks of the reduced and full design matrices, respectively. When each model matrix has full column rank, these are also the numbers of estimated coefficients, including an intercept if present. The number of independent added terms is \(q=p_F-p_R\), which must be positive. The reduced and full models must use the same \(n\) response observations.

The conditional wording matters: the test asks whether the added block improves the model after the reduced-model terms are included. It does not test whether the response is associated with each added term in isolation, and it does not establish a causal effect.

Deriving the Partial F Statistic

For each model, least squares chooses coefficients to minimize the residual sum of squares. Denote the reduced- and full-model residual sums of squares by \(SSE_R\) and \(SSE_F\). Because the full model includes every fit available to the reduced model and possibly more, \(SSE_F\leq SSE_R\). The reduction in residual sum of squares is \(SSE_R-SSE_F\). Under the null, this reduction reflects error variation captured by the extra terms rather than a genuine change in the mean structure.

The partial F statistic compares the reduction per added degree of freedom with the full model’s residual mean square. If the errors are independent and normally distributed with a common variance, and the null hypothesis holds, each quantity estimates the same error variance after scaling by its degrees of freedom. Their ratio then has an \(F\) distribution.

The numerator is the mean square for the added block: the reduction in residual sum of squares divided by the number of independent added terms. The denominator is the full model’s residual mean square, \(SSE_F/(n-p_F)\). Large positive values of \(F\) indicate that the added block accounts for a substantial reduction relative to the remaining residual variation.

Why the Reference Distribution Is an F Distribution

The result can be justified using projection matrices. For an ordinary least-squares model, let \(P_R\) and \(P_F\) be the matrices that project a response vector onto the column spaces of \(X_R\) and \(X_F\). These are symmetric and idempotent. Since the model spaces are nested, \(P_FP_R=P_R\). The fitted values are \(P_Ry\) and \(P_Fy\), and the residual sums of squares are \(y'(I-P_R)y\) and \(y'(I-P_F)y\).

Subtracting the two residual sums of squares gives the reduction attributable to the added model space:

The matrix \(P_F-P_R\) is symmetric and idempotent. Its rank is \(p_F-p_R=q\), because it projects onto the part of the full model space that is perpendicular to the reduced model space. Under \(H_0\), the true mean vector lies in the reduced model space, so \((P_F-P_R)X_R\beta_R=0\). With \(y=X_R\beta_R+\varepsilon\), the reduction therefore depends only on the errors:

For the full-model residual sum of squares, \(I-P_F\) is also a symmetric idempotent projection, now of rank \(n-p_F\). Under the same null model,

These two quadratic forms are independent: their projection matrices act on perpendicular subspaces, since \((P_F-P_R)(I-P_F)=0\). With normally distributed errors, orthogonal projections of the error vector are independent. Dividing each chi-square quantity by its degrees of freedom and taking their ratio cancels \(\sigma^2\), yielding the stated \(F_{q,n-p_F}\) distribution under \(H_0\). This derivation also shows why normality and the common error variance are needed for the exact small-sample reference distribution.

Assumptions for the Test

The partial F test is exact under the classical normal linear model assumptions for the full model, together with the nesting and same-observations requirements. State and assess these conditions before relying on an F reference distribution.

• Correct nesting and a specified added block. The reduced-model space must be contained in the full-model space, and the tested \(q\) independent terms must be identified in advance. Both models must use the same response and observations.
• Linear mean structure in the model coefficients. The response mean is represented by the stated design matrix and coefficients. Predictors and transformations may be nonlinear in their raw values, but their coefficients enter linearly.
• Zero-mean errors and constant variance. Conditional on the predictors, \(E(\varepsilon)=0\) and \(\operatorname{Var}(\varepsilon)=\sigma^2 I\). In particular, errors have a common variance and are uncorrelated.
• Normal errors for the exact F distribution. Conditional on the design, \(\varepsilon\sim N(0,\sigma^2 I)\). Without normality, the same statistic may sometimes be used as an approximation in sufficiently large samples, but its exact finite-sample F distribution is not guaranteed.
• Estimable models and positive residual degrees of freedom. The model ranks must be well defined, \(q>0\), and \(n-p_F>0\), so the full-model residual mean square can be calculated.

Residual plots and knowledge of how the data were collected can help assess whether the mean structure, independence, and variance conditions are plausible. A numerical F statistic does not repair a violated assumption. Also, “independent errors” concerns the errors conditional on the model predictors; it is not a claim that the observed responses themselves must be independent of every predictor.

Common Mistakes and Practical Meaning

• Using the wrong degrees of freedom. The numerator degrees of freedom are the rank increase \(p_F-p_R\), not the total number of full-model coefficients. The denominator degrees of freedom are \(n-p_F\).
• Comparing models fitted to different observations. Different samples can change the residual sums of squares for reasons unrelated to the added terms. The standard partial F formula requires the same response observations in both fits.
• Testing coefficients one at a time instead of as a block. A partial F test assesses the joint null for all added coefficients. It does not imply that each term is individually important or unimportant.
• Treating any decrease in SSE as evidence. A nested full model cannot have a larger least-squares SSE, but some decrease is expected just from adding terms. The test measures that decrease relative to the estimated error variance and the number of added degrees of freedom.
• Reading a p-value as the probability that the null is true. The p-value is the probability, assuming the null and test assumptions, of obtaining an F statistic at least as large as the observed one. It is not the probability that the null is true.

The partial F test is useful when the scientific or modeling question concerns a block of terms rather than one coefficient. Its value is not a mechanical instruction to enlarge a model whenever the p-value is small. The terms should make sense for the modeling question, and conclusions should be considered alongside residual evidence and the limitations of the assumed regression model.