Testing Added Regressors With a Worked Example

Why Test a Block of Added Regressors?

A regression analysis may begin with a baseline model and then add several regressors representing a clinical factor, a group of indicators, or a set of interaction terms. The relevant question is whether the added block contributes to the fitted mean after accounting for the regressors already in the baseline model. Testing the block jointly answers that question; it is not the same as testing each added coefficient separately.

The partial F test compares the residual sums of squares from two nested fits. Its numerator measures the reduction in residual variation when the block is included, adjusted for the number of independent directions added. Its denominator estimates the error variance using the unrestricted model.

Definition: Let $X_R$ be the design matrix for a restricted model and $X_U$ the design matrix for an unrestricted model, with $\mathcal{C}(X_R)\subseteq\mathcal{C}(X_U)$. Write $r_R=\operatorname{rank}(X_R)$, $r_U=\operatorname{rank}(X_U)$, $q=r_U-r_R$, and $\nu=n-r_U$. The restricted and unrestricted residual sums of squares are $RSS_R=\|Y-P_RY\|^2$ and $RSS_U=\|Y-P_UY\|^2$, where $P_R$ and $P_U$ are the orthogonal projection matrices onto the respective model spaces. When $q>0$ and $\nu>0$, the partial F statistic is $$ F=\frac{(RSS_R-RSS_U)/q}{RSS_U/\nu}. $$

The null hypothesis says that the mean vector belongs to the restricted model space: the added regressors do not enlarge the mean needed to describe the data. In a full-rank parameterization, this is equivalent to setting all $q$ added coefficients to zero. More generally, it is the model-space comparison that defines the test, so the rank difference $q$, not simply the number of columns written in $X_U$, determines the numerator degrees of freedom.

Exact Calibration and Its Conditions

For the usual exact normal-theory test, assume a fixed design and the model $Y=\mu+\varepsilon$, where $\varepsilon\sim N(0,\sigma^2 I_n)$, $\sigma^2>0$. Under the null, $\mu\in\mathcal{C}(X_R)$. The model spaces must be nested, and both $q>0$ and $\nu>0$ are required: there must be at least one added direction and at least one unrestricted residual degree of freedom.

Theorem: Under these conditions and the null hypothesis, $F\sim F(q,\nu)$. Thus the exact upper-tail p-value for an observed value $f$ is $P(F_{q,\nu}\ge f)$, where $F_{q,\nu}$ denotes a random variable with the $F(q,\nu)$ distribution.

Proof. Since the model spaces are nested, $P_U-P_R$ is an orthogonal projection onto the part of the unrestricted space orthogonal to the restricted space. It has rank $q$. Also, $I-P_U$ is an orthogonal projection of rank $\nu$, and its image is orthogonal to the image of $P_U-P_R$. The difference in residual sums of squares can be written

RSS_R-RSS_U=Y^\mathsf{T}(P_U-P_R)Y.

Under the null, $(P_U-P_R)\mu=0$ and $(I-P_U)\mu=0$. Therefore,

\frac{RSS_R-RSS_U}{\sigma^2} =\frac{\varepsilon^\mathsf{T}(P_U-P_R)\varepsilon}{\sigma^2} \sim\chi^2_q, \qquad \frac{RSS_U}{\sigma^2} =\frac{\varepsilon^\mathsf{T}(I-P_U)\varepsilon}{\sigma^2} \sim\chi^2_\nu.

These chi-square variables are independent. Indeed, the two projected error vectors are jointly normal, and their cross-covariance is $\sigma^2(P_U-P_R)(I-P_U)=0$; jointly normal vectors with zero cross-covariance are independent. Dividing each chi-square variable by its degrees of freedom and taking their ratio gives the stated $F(q,\nu)$ distribution. Since $\nu>0$ and $\sigma^2>0$, the unrestricted residual sum of squares is positive with probability one. $\square$

Worked Calculations

The examples use orthonormal design directions $e_1,\ldots,e_n$, so each least-squares residual sum of squares can be checked by summing the squared response coordinates outside the model space. This makes the model comparison and the arithmetic explicit.

Worked Example: Testing Two Added Regressors

Suppose $n=10$, the restricted model space is spanned by $e_1,e_2$, and the unrestricted model adds $e_3,e_4$. Thus $r_R=2$, $r_U=4$, and the null hypothesis is that the mean needs no component in either added direction. Take $Y=e_1+2e_2+3e_3+3e_4+\sqrt{2}(e_5+e_6+e_7+e_8+e_9+e_{10})$.

For the restricted fit, the residual coordinates are $3,3,\sqrt2,\sqrt2,\sqrt2,\sqrt2,\sqrt2,\sqrt2$, so $RSS_R=3^2+3^2+6(\sqrt2)^2=9+9+12=30$. For the unrestricted fit, the $e_3$ and $e_4$ components are fitted as well, leaving only the last six coordinates: $RSS_U=6(\sqrt2)^2=12$.

The numerator degrees of freedom are $q=4-2=2$, and the denominator degrees of freedom are $\nu=10-4=6$. Therefore,

F=\frac{(30-12)/2}{12/6} =\frac{9}{2} =4.5.

For $F(2,6)$, the upper-tail probability at $f$ is $\left(6/(6+2f)\right)^3$. Hence the p-value is $\left(6/(6+2(4.5))\right)^3=(6/15)^3=0.064$. At significance level $0.05$, this is not enough evidence to reject the joint null hypothesis.

Worked Example: Testing One Added Regressor

Let $n=5$, with restricted space spanned by $e_1,e_2$ and unrestricted space spanned by $e_1,e_2,e_3$. Use $Y=e_1+2e_2+\sqrt6e_3+\sqrt2(e_4+e_5)$. The null says that the mean does not require the added $e_3$ direction.

The restricted fit leaves the $e_3,e_4,e_5$ coordinates, giving $RSS_R=(\sqrt6)^2+(\sqrt2)^2+(\sqrt2)^2=6+2+2=10$. The unrestricted fit also captures the $e_3$ coordinate, so $RSS_U=2+2=4$. Here $q=3-2=1$ and $\nu=5-3=2$, both positive. The statistic is

F=\frac{(10-4)/1}{4/2}=\frac{6}{2}=3.

For $F(1,2)$, the upper-tail probability at $f$ is $1-\sqrt{f/(f+2)}$. Thus $p=1-\sqrt{3/(3+2)}=1-\sqrt{3/5}\approx 0.2254$. The observed reduction in residual sum of squares is not unusual enough under the null to reject at the $0.05$ level.

Worked Example: A Small Reduction From a Two-Regressor Block

Return to $n=10$, with restricted space spanned by $e_1,e_2$ and unrestricted space spanned by $e_1,e_2,e_3,e_4$. Let $Y=e_1+2e_2+e_3+e_4+\sqrt2(e_5+e_6+e_7+e_8+e_9+e_{10})$.

The restricted model leaves the $e_3,e_4$ coordinates and the last six coordinates, so $RSS_R=1^2+1^2+6(\sqrt2)^2=1+1+12=14$. The unrestricted model captures the $e_3,e_4$ coordinates, leaving $RSS_U=6(\sqrt2)^2=12$. Thus $q=4-2=2$, $\nu=10-4=6$, and

F=\frac{(14-12)/2}{12/6}=\frac{1}{2}=0.5.

The exact p-value is $\left(6/(6+2(0.5))\right)^3=(6/7)^3=216/343\approx0.6297$. This small reduction in residual sum of squares, relative to the unrestricted residual variance and the two added directions, gives little evidence against the block null hypothesis.

A Reliable Calculation Sequence

The same sequence applies whether the added block contains one regressor or several. The key is to identify the two nested fits before using any formula.

Specify the comparison.
Name the restricted model and the unrestricted model, and state the null that the mean lies in the restricted model space.

Check nesting and ranks.
Confirm $\mathcal{C}(X_R)\subseteq\mathcal{C}(X_U)$. Calculate $q=r_U-r_R$ and $\nu=n-r_U$; require $q>0$ and $\nu>0$.

Obtain both residual sums of squares.
Use the same response and observations in both fits. Nesting implies $RSS_R\ge RSS_U$, apart from numerical rounding.

Calculate and calibrate.
Compute $(RSS_R-RSS_U)/q$, $RSS_U/\nu$, their ratio, and the upper-tail probability from $F(q,\nu)$.

Interpretation and Common Pitfalls

The numerator is the reduction in residual sum of squares per added model-space direction; the denominator is the unrestricted residual mean square. A large statistic indicates that the reduction is large compared with the remaining residual variation. The p-value measures how often an $F(q,\nu)$ statistic would be at least as large under the null and the stated normal-theory assumptions. It does not measure the probability that the null hypothesis is true.

A common mistake is to count added columns and use that number as $q$ without checking their independence from the existing design. If some added columns lie in the restricted space or are redundant with one another, the rank increase can be smaller than the column count. The correction is to compute $q=r_U-r_R$. Another invalid calculation occurs when $q=0$: the model spaces are then equal, so there is no added direction and the partial F statistic is undefined. It is not an $F$ test with zero numerator degrees of freedom.

Also check that the two fits are genuinely nested and use the same observations. If they are not nested, $RSS_R-RSS_U$ need not represent a nonnegative reduction attributable to the added block, and the partial F formula does not apply. Finally, the exact $F(q,\nu)$ calibration depends on independent normal errors with common variance and a fixed design; without those assumptions, this exact result is not justified.

Key takeaway: For a valid exact partial F test, compare nested model spaces, use $q=r_U-r_R>0$ and $\nu=n-r_U>0$, and calculate $\{(RSS_R-RSS_U)/q\}/(RSS_U/\nu)$. Interpret its upper-tail p-value under the specified normal linear model.

Check Your Understanding

For each question, identify the relevant model comparison or calculation.

A restricted model has rank 3, an unrestricted model has rank 6, and $n=15$. What are $q$ and $\nu$?
Why must the design spaces be nested for the usual partial F comparison?
If $RSS_R=48$, $RSS_U=30$, $q=3$, and $\nu=10$, calculate the partial F statistic.
What happens to the test definition if the restricted and unrestricted model spaces have the same rank and are nested?
State the error assumptions that give the partial F statistic its exact null distribution.