Comparing Linear and Quadratic Fits

Why Compare the Fits?

A quadratic regression can fit the observed data at least as closely as a linear regression, because the linear form is included as a special case of the quadratic form. But a smaller residual sum of squares alone does not show whether the improvement is more than would be expected from adding another coefficient. A partial F-test addresses that question by comparing the two fits on the same observations.

The test concerns evidence for an additional quadratic term. It does not decide on its own whether a curved model is useful: the size and consequences of the change in fitted values also matter. The result should therefore be read in two parts: first, what the data say statistically about adding curvature; second, whether the resulting curvature matters for the purpose of the analysis.

The Models and the Comparison

Suppose the response $Y_i$ and predictor value $x_i$ are observed for the same $n$ cases. The reduced, linear model and the full, quadratic model are

\begin{aligned} Y_i &= \beta_0+\beta_1x_i+\varepsilon_i &&\text{(linear model)},\\ Y_i &= \beta_0+\beta_1x_i+\beta_2x_i^2+\varepsilon_i &&\text{(quadratic model)}. \end{aligned}

The linear model is nested in the quadratic model: setting $\beta_2=0$ in the quadratic model gives the linear model. Thus, the models differ by one added parameter, the coefficient of $x_i^2$. For both fits, use the same response observations, predictor values, and least-squares fitting method. Let $RSS_L$ be the linear model’s residual sum of squares and $RSS_Q$ the quadratic model’s residual sum of squares.

Definition: The partial F-test for comparing a linear and quadratic fit tests whether adding the quadratic term reduces residual variation enough to provide evidence against the linear model. Its null hypothesis is $H_0:\beta_2=0$; its alternative is $H_a:\beta_2\ne 0$. The test compares the reduction $RSS_L-RSS_Q$ with the residual variation remaining in the quadratic fit.

The full quadratic design matrix must have rank 3, meaning that its columns $1$, $x$, and $x^2$ are linearly independent. For this model, that requires at least three distinct observed predictor values. Also require $n>3$, so that the full model has positive residual degrees of freedom. Under these conditions, the quadratic model has $n-3$ residual degrees of freedom, while the linear model has $n-2$. The added term accounts for one numerator degree of freedom.

Theorem: If the full design matrix has rank 3, $n>3$, and the errors are independent with mean 0, common variance $\sigma^2$, and a normal distribution, then under $H_0$, $$ F=\frac{(RSS_L-RSS_Q)/1}{RSS_Q/(n-3)} $$ has an $F$ distribution with 1 and $n-3$ degrees of freedom. The numerator measures the reduction in residual sum of squares per added parameter; the denominator is the quadratic model’s residual mean square.

Why the Partial F-Statistic Has This Form

Adding a term cannot make the least-squares residual sum of squares larger: the quadratic model can always reproduce any linear fit by choosing $\beta_2=0$. Therefore $RSS_L-RSS_Q\ge 0$. The test asks whether this reduction is large relative to the residual variation left after fitting the quadratic model.

A short derivation explains both the numerator degrees of freedom and the reference distribution. Treat the observed predictor values as fixed. Let $P_L$ and $P_Q$ be the projection matrices onto the spaces of fitted values from the linear and quadratic models. The linear fitted-value space is contained in the quadratic one. Consequently, $P_Q-P_L$ is a projection onto the one added direction, and its rank is $3-2=1$. The quadratic residual projection $I-P_Q$ has rank $n-3$; these two projected spaces are orthogonal.

Under $H_0$, the mean response lies in the linear fitted-value space. Normal errors projected onto the added direction and onto the quadratic residual space therefore give independent sums of squares:

\frac{RSS_L-RSS_Q}{\sigma^2}\sim\chi^2_1, \qquad \frac{RSS_Q}{\sigma^2}\sim\chi^2_{n-3}.

Dividing each quantity by its degrees of freedom and taking their ratio cancels $\sigma^2$, giving the stated $F_{1,n-3}$ distribution. The normality assumption gives this exact small-sample result; independence and constant variance are also part of the model conditions.

Calculating and Interpreting the Test

For a chosen significance level $\alpha$, calculate the statistic and its upper-tail p-value from the $F_{1,n-3}$ distribution. A small p-value means that, if the linear model and the stated error assumptions hold, a reduction in residual sum of squares at least this large would be unusual. It is evidence against $H_0$, not the probability that $H_0$ is true.

Worked Example: A Borderline Improvement in a Growth Model

In a hypothetical experiment, a researcher records a growth-rate response, in milligrams per litre per day, at several temperature settings. The same $n=12$ observations are used for both fits, and there are at least three distinct temperatures. Suppose the linear fit has $RSS_L=240$, while the quadratic fit has $RSS_Q=150$. Test at $\alpha=0.05$.

The full design has rank 3 and $12>3$, so the test has numerator degrees of freedom 1 and denominator degrees of freedom $12-3=9$. The reduction is $240-150=90$. Thus,

F=\frac{(240-150)/1}{150/9} =\frac{90}{16.6667} =5.400.

For an $F_{1,9}$ distribution, the upper-tail p-value is approximately 0.045. Since $0.045<0.05$, reject $H_0$ at the 5% level. These data provide evidence that the quadratic term improves the fit beyond what would be expected from adding a term when the linear model is adequate. This result does not establish that temperature causes the observed response pattern, nor does it by itself establish that the amount of curvature is practically important.

Statistical Evidence and Practical Usefulness

A p-value addresses compatibility with a specified null model, not the importance of the estimated curvature. With a large sample, a small reduction in residual variation may be statistically detectable. Conversely, a potentially meaningful curve may not be detected clearly in a small or noisy sample. After the test, examine the fitted curves and their predictions across the observed predictor range, taking account of the response’s units and the purpose of the analysis.

Worked Example: Significant but Possibly Negligible Curvature

A hypothetical quality-control analysis relates machine speed $x$, in units of 10 rotations per minute, to a measured response in millimetres. There are $n=500$ observations and at least three distinct speed values. The linear fit has $RSS_L=1000$, and the quadratic fit has $RSS_Q=990$. The quadratic fit’s largest change in fitted response relative to the linear fit anywhere in the observed speed range is reported as 0.04 mm; that difference is small compared with the typical response variation in this application.

The full model has $500-3=497$ residual degrees of freedom. The partial F-statistic is

F=\frac{(1000-990)/1}{990/497} =\frac{10}{1.99195} =5.020.

For an $F_{1,497}$ distribution, the upper-tail p-value is approximately 0.02549. At $\alpha=0.05$, this is evidence against a strictly linear mean pattern under the model assumptions. But the improvement corresponds to a very small reported change in fitted response over the observed range. If differences of 0.04 mm would not affect a quality decision, the added curvature may not be useful for that decision. Statistical evidence and practical value are related but distinct judgments.

The $R^2$ value also cannot replace the partial F-test. Because the models use the same data and the quadratic model contains the linear one, its unadjusted $R^2$ cannot be lower. That increase alone does not account for the added parameter or the amount of residual variation. The partial F-statistic makes that comparison explicitly.

What a Nonsignificant Result Means

A large p-value is not evidence that the true mean relationship is exactly linear. It says that the observed reduction in residual sum of squares is not sufficiently unusual under $H_0$ to reject the linear model at the chosen significance level. The data may be too noisy or too limited to distinguish the models. Conclusions should be framed as a lack of sufficient evidence for added curvature, not as proof that curvature is absent.

Worked Example: A Nonsignificant Reduction

In a hypothetical analysis of study time $x$, in hours, and exam score $Y$, in points, suppose $n=18$, at least three distinct study times are observed, $RSS_L=84.0$, and $RSS_Q=78.9$. Use $\alpha=0.05$.

The full model has $18-3=15$ residual degrees of freedom, and the reduction is $84.0-78.9=5.1$. Therefore,

F=\frac{(84.0-78.9)/1}{78.9/15} =\frac{5.1}{5.26} =0.9696.

The upper-tail p-value from an $F_{1,15}$ distribution is approximately 0.34. Because this exceeds 0.05, do not reject $H_0$. The analysis does not provide sufficient evidence that adding the quadratic term improves the fit. It does not prove that the relationship between study time and score is linear, and it does not imply that the quadratic residual sum of squares should equal the linear one.

Assumptions and Common Pitfalls

Before relying on the exact partial F-test, verify that the comparison is genuinely nested and uses the same observations and response in both fits. Check that there are at least three distinct predictor values and that $n>3$. The error conditions for the stated exact result are independent errors, a mean of zero, constant variance, and normality. In a real analysis, assess whether these assumptions are plausible for the data; the test cannot repair a poor model or an unsuitable sampling design.

Using different observations for the two fits. Then the residual sums of squares do not describe a direct nested comparison. Fit both models to the same cases.
Calling any decrease in RSS evidence for curvature. The quadratic model’s RSS cannot increase, even when the additional term is not useful. Compare the reduction with the quadratic model’s residual mean square using the partial F-statistic.
Ignoring rank and degrees of freedom. The formula with denominator degrees of freedom $n-3$ requires a rank-3 full design and $n>3$. If there are fewer than three distinct $x_i$ values, the three polynomial columns are not full rank, so this stated test does not apply as written.
Treating a nonsignificant p-value as proof of a straight-line relationship. A nonsignificant result is a failure to find sufficient evidence against $H_0$ at the chosen level, not proof that $H_0$ is true.
Treating significance as a practical recommendation. A small p-value does not say whether the change in predictions is large enough to matter. Interpret the fitted differences in the outcome’s units and in the analysis context.

Key takeaway: The partial F-test compares the reduction $RSS_L-RSS_Q$ with the quadratic model’s residual variation, using one numerator degree of freedom and $n-3$ denominator degrees of freedom when the full design has rank 3 and $n>3$. Its p-value addresses evidence for curvature; usefulness depends on the size and consequences of the fitted change.

Check Your Understanding

Answer each question using the model comparison and conditions described above.

Write the null and alternative hypotheses for comparing a linear mean model with a quadratic mean model.
For $n=24$ and a rank-3 quadratic design, what are the numerator and denominator degrees of freedom for the partial F-test?
Why can the quadratic model’s residual sum of squares not be greater than the linear model’s residual sum of squares when both are fit to the same data by least squares?
A test gives $p=0.12$ at $\alpha=0.05$. State a correct conclusion and one conclusion that would be too strong.
Why might statistically significant evidence for curvature still fail to justify using the quadratic model for a practical decision?