Deriving the Partial F Statistic

Why the Partial F Statistic Uses Two Mean Squares

When a full regression model adds predictors to a reduced model, its residual sum of squares cannot increase if both models are fitted to the same outcome and observations. But a decrease by itself does not say whether the added terms improved fit substantially: adding more parameters can reduce residual error even when those terms do not describe a real signal. The partial F statistic puts the reduction on a scale that accounts for both the number of added parameters and the residual variation left in the full model.

The statistic compares two quantities. Its numerator is the reduction in residual sum of squares divided by the number of added parameters. Its denominator is the full model’s residual mean square: residual sum of squares divided by the full model’s residual degrees of freedom. Under the null hypothesis and the usual normal linear model assumptions, this ratio has an F distribution.

Models, Residual Sums of Squares, and Degrees of Freedom

Suppose the reduced model and full model use the same \(n\) observations and the same response variable. The full model is nested in the sense that it contains every term in the reduced model, possibly with additional terms. Let \(RSS_R\) denote the reduced model’s residual sum of squares and \(RSS_F\) the full model’s residual sum of squares. The subscripts \(R\) and \(F\) stand for reduced and full.

Let \(p_R\) and \(p_F\) be the numbers of parameters estimated in the reduced and full models, respectively. Count all estimated regression coefficients, including an intercept when the model has one. The number of added parameters is \(q=p_F-p_R\). The full model must add at least one parameter, so \(q>0\). If the full model has full column rank, its residual degrees of freedom are \(n-p_F\).

The numerator, \((RSS_R-RSS_F)/q\), is the average reduction in residual sum of squares per added parameter. The denominator, \(RSS_F/(n-p_F)\), is the full model’s residual mean square, often used to estimate the common error variance. Both quantities are measured in squared units of the response, so their ratio is dimensionless.

The statistic is defined only when the models are genuinely nested, \(q>0\), and the full model has positive residual degrees of freedom. The reduced and full residual sums of squares must also come from the same observations and response. Comparing residual sums of squares from different samples does not give this statistic a valid nested-model interpretation.

Why These Degrees of Freedom Appear

The numerator degrees of freedom count the additional independent directions in which the full model can fit the data. With \(q\) added parameters, the expanded model has \(q\) more fitted dimensions than the reduced model. The denominator degrees of freedom count what remains for estimating residual variation after fitting the full model: \(n-p_F\).

The F distribution arises from a ratio of two independent, variance-scaled sums of squares. To see why, write the regression model for the full set of observations as \(Y=\mu+\varepsilon\), where \(Y\) is the vector of responses and \(\mu\) is its mean vector. Under the null hypothesis, the reduced model describes the mean: there is no additional mean component in the directions introduced by the full model. Assume the design values are fixed and that \(\varepsilon\sim N(0,\sigma^2 I)\), so the errors are independent, normally distributed with mean zero and common variance \(\sigma^2\). Also assume the model matrices have full column rank.

Fitting each model projects the response onto its model space. Because the reduced model’s space is contained in the full model’s space, the added directions form a \(q\)-dimensional space perpendicular to the reduced model’s space. Under the null, the reduction \(RSS_R-RSS_F\) is the squared length of the error projected onto those added directions. The remaining full-model residuals lie perpendicular to the full model’s space, a space with dimension \(n-p_F\).

For a normal error vector with covariance matrix \(\sigma^2 I\), projections onto perpendicular spaces are independent. The squared length of the error projected onto a space of dimension \(d\), divided by \(\sigma^2\), has a chi-square distribution with \(d\) degrees of freedom. Therefore, under the null,

and these two quantities are independent. Dividing each by its degrees of freedom gives independent estimates on the same variance scale:

By definition, the ratio of two independent chi-square variables each divided by its degrees of freedom has an F distribution with those numerator and denominator degrees of freedom. The common \(\sigma^2\) cancels in the ratio, giving the reference distribution for the partial F statistic:

Worked Examples: Reading the Formula Carefully

How to Use the Statistic, and What It Does Not Say

Under the null, the numerator and denominator estimate the same error variance, so their ratio is typically near 1, though sampling variation can make it larger or smaller. A large value means the average reduction in residual sum of squares per added parameter is large relative to the full model’s residual mean square. For a conventional upper-tail test, unusually large values count against the null because adding the terms has produced a reduction large relative to the residual variation.

This scale comparison does not by itself describe the practical importance of the added predictors. The F statistic has no units and depends on the amount of information in the data, the number of added parameters, and residual variation. A small change in fit can yield evidence against the null in a sufficiently informative sample, while a potentially important change can be estimated imprecisely. The statistic is an inferential tool, not a measure of practical value or evidence of causation.

The normality and equal-variance assumptions support the exact F reference distribution derived above. The nesting and same-sample conditions are also essential to the sum-of-squares comparison. If these conditions are not met, the displayed statistic may still be calculable, but its stated exact null distribution is not justified by this derivation.

Common Mistakes

• Using the wrong numerator degrees of freedom. The numerator degrees of freedom are the number of parameters added, \(p_F-p_R\), not the number of parameters in the full model.
• Using the reduced model for the denominator. The denominator is the full model residual mean square, \(RSS_F/(n-p_F)\), because it estimates residual variance after fitting the full model.
• Forgetting to divide the improvement by \(q\). The numerator is an average reduction per added parameter, not simply \(RSS_R-RSS_F\).
• Comparing fits from different samples. The partial F formula requires nested models fit to the same response observations; otherwise the RSS difference does not isolate the added model dimensions.
• Calling a larger F value proof that the added predictors are important. It indicates a larger fit improvement relative to residual variation, but statistical evidence and practical importance are distinct judgments.