Worked Example: Selecting Polynomial Degree

Introduction

A polynomial can represent a curved relationship between a predictor and a response. Increasing its degree gives the fitted model more flexibility, but flexibility alone is not a reason to prefer a higher-degree model. The practical question is whether a candidate degree predicts well on observations that were not used to fit it.

This tutorial works through that decision using training and validation results. The validation loss selects the degree; a separate test set is reserved to assess the selected model after the choice is made. The examples use either mean squared error or mean absolute error, so the stated loss—not an assumed default—determines the comparison.

Definition: A polynomial regression model of degree $d$ represents the conditional mean of a response $Y$, given a predictor value $x$, as $$ E[Y\mid x]=\beta_0+\beta_1x+\beta_2x^2+\cdots+\beta_dx^d. $$ The degree $d$ is the largest power included. A degree-zero model has an intercept only; a degree-one model is linear. A fitted model uses estimated coefficients to produce a prediction $\hat y_j$ for an observation with predictor value $x_j$. The residual for evaluating a prediction is $e_j=y_j-\hat y_j$.

What the Loss Measures

For $n$ observations in a specified data set, mean squared error (MSE) is the average of the squared prediction errors. It measures the mean squared distance between observed responses and predictions; it is nonnegative, and smaller values indicate better predictions according to this criterion.

\operatorname{MSE}=\frac{1}{n}\sum_{j=1}^{n}(y_j-\hat y_j)^2 =\frac{1}{n}\sum_{j=1}^{n}e_j^2

Mean absolute error (MAE) instead averages the absolute prediction errors:

\operatorname{MAE}=\frac{1}{n}\sum_{j=1}^{n}|y_j-\hat y_j| =\frac{1}{n}\sum_{j=1}^{n}|e_j|

The response scale and the decision goal help determine which loss is appropriate. MSE penalizes a large error more heavily because it squares the error; MAE gives errors a penalty proportional to their absolute size. For example, an error of 4 contributes 16 to the MSE numerator but 4 to the MAE numerator. Do not compare an MSE for one degree with an MAE for another as though they were the same criterion.

Justification: The displayed MSE formula is the empirical average of the chosen squared-error loss. Each observation contributes its own loss, $e_j^2$; summing these contributions gives total loss, and dividing by $n$ gives loss per observation. Thus, when candidates are evaluated on the same observations, the degree with the smaller MSE has the smaller average squared prediction error on that data set. This arithmetic does not establish that its future prediction error will also be smaller; validation and test observations address that question.

Training loss is calculated on the observations used to fit each candidate. Validation loss is calculated on separate observations and is used to select among candidates. For least-squares polynomial fits with an intercept on the same training observations, the candidate model spaces are nested: every polynomial of degree $d$ is also representable at degree $d+1$, by setting the additional coefficient to zero. Consequently, the minimum training sum of squared errors cannot increase as degree increases. Training loss can therefore keep improving even when a more flexible model does not predict better on validation data.

Validation loss need not decrease with degree. A common procedure is to select the candidate with the smallest validation loss for the stated criterion. If multiple candidates tie, use a tie-breaking rule specified in advance, such as choosing the lower degree. Once the degree is selected, the test set is used for final assessment, not to choose a different degree.

Worked Selection Process

State the candidates and loss.
List the polynomial degrees to compare and specify the evaluation criterion, such as MSE or MAE.

Calculate each validation loss.
Use the residuals for that degree on the same validation observations. Average the losses; do not compare totals when the number of observations differs.

Choose the degree.
Select the lowest validation loss. Apply any prespecified tie rule rather than inventing a new rule after seeing the results.

Assess once on test data.
After the choice is fixed, use the untouched test set to report the final loss. Do not use that result to revise the degree and still call it an independent final assessment.

Worked Examples

Worked Example: Selecting a Degree by Validation MSE

A hypothetical analyst predicts a numerical response from one predictor and fits polynomial degrees 1, 2, and 3 by least squares. The stated selection loss is validation MSE. The following residuals are supplied; each degree has four training residuals and three validation residuals. A separate test set has four observations. This is a prediction exercise, not a cash-flow valuation: no interest rate or mortality assumption is involved, and no probability distribution or distribution parameters are specified.

Degree	Training residuals	Validation residuals
1	2, −1, 1, −2	2, −2, 2
2	1, 0, −1, 1	1, −2, 0
3	0.5, −0.5, 0, −1	2, −2, 1

Step 1: Calculate training MSE. For degree 1, the squared residuals sum to $2^2+(-1)^2+1^2+(-2)^2=4+1+1+4=10$, so training MSE is $10/4=2.5$. For degree 2, the sum is $1^2+0^2+(-1)^2+1^2=1+0+1+1=3$, giving $3/4=0.75$. For degree 3, the sum is $0.5^2+(-0.5)^2+0^2+(-1)^2=0.25+0.25+0+1=1.5$, giving $1.5/4=0.375$.

Step 2: Calculate validation MSE. Degree 1 has squared residual sum $2^2+(-2)^2+2^2=4+4+4=12$, so its validation MSE is $12/3=4$. Degree 2 has sum $1^2+(-2)^2+0^2=1+4+0=5$, so its validation MSE is $5/3=1.6667$ to four decimal places. Degree 3 has sum $2^2+(-2)^2+1^2=4+4+1=9$, so its validation MSE is $9/3=3$.

Step 3: Select the degree. The smallest validation MSE is $5/3$, so select degree 2. The decreasing training MSEs do not change that decision: degree 3 has lower training MSE than degree 2, but higher validation MSE.

Step 4: Assess the selected model on the test set. The degree-2 model has test residuals $1,0,-1,2$. Its test MSE is $(1^2+0^2+(-1)^2+2^2)/4=(1+0+1+4)/4=6/4=1.5$. Report this as the final test assessment; do not use it to reconsider the degree. The method avoids choosing the more complex degree merely because it fits training data better.

Worked Example: A Tie Under Validation MAE

A hypothetical service team predicts repair time, measured in hours, from equipment age. It compares degrees 1, 2, and 3. The stated goal is to minimize validation MAE, and the prespecified tie rule is to select the lower degree. The residuals below are supplied for four training and four validation observations. This is not an interest or mortality calculation; no probability distribution or parameters are assumed.

Degree	Training residuals	Validation residuals
1	2, −2, 2, −2	3, −1, 2, −2
2	1, −1, 1, −1	1, −2, 1, −2
3	0.5, −0.5, 0.5, −0.5	0, 0, 0, 6

Step 1: Check training MAE. Degree 1 has absolute-residual sum $2+2+2+2=8$, hence MAE $8/4=2$. Degree 2 has sum $1+1+1+1=4$, hence MAE $4/4=1$. Degree 3 has sum $0.5+0.5+0.5+0.5=2$, hence MAE $2/4=0.5$. These values describe fit on training observations, not the selection result.

Step 2: Calculate validation MAE. For degree 1, the absolute-residual sum is $3+1+2+2=8$, so MAE is $8/4=2$. For degree 2, the sum is $1+2+1+2=6$, so MAE is $6/4=1.5$. For degree 3, the sum is $0+0+0+6=6$, also giving $6/4=1.5$.

Step 3: Apply the tie rule. Degrees 2 and 3 tie for the lowest validation MAE. The stated rule selects the lower degree, so choose degree 2. The rule does not claim that degree 2 must be better in every future sample; it makes the decision reproducible without using test results to break the tie.

Step 4: Make the final assessment. Suppose the selected degree-2 model has test residuals $-1,0,2$. The test MAE is $(|-1|+|0|+|2|)/3=(1+0+2)/3=1$ hour. This reports its test performance in the response units. The calculation avoids incorrectly selecting degree 3 just because its training MAE is smallest.

Worked Example: Refit the Selected Degree, Then Test

A hypothetical analyst predicts daily energy demand from a temperature measure. Candidate degrees 1, 2, and 3 are fit on four training observations; their validation residuals on three separate observations are provided below. Selection uses validation MSE, with no tie. After selection, the analyst refits that degree using the training and validation observations together, then evaluates the refitted model on a separate test set. No interest rate or mortality assumption is relevant; no probability distribution or distribution parameters are specified.

Degree	Training MSE (reported)	Validation residuals
1	4	3, −1, 1
2	1	1, 1, −1
3	0.25	0, 2, −2

Step 1: Calculate validation MSEs. Degree 1 has squared-residual sum $3^2+(-1)^2+1^2=9+1+1=11$, so its validation MSE is $11/3=3.6667$ to four decimal places. Degree 2 has sum $1^2+1^2+(-1)^2=1+1+1=3$, giving $3/3=1$. Degree 3 has sum $0^2+2^2+(-2)^2=0+4+4=8$, giving $8/3=2.6667$ to four decimal places.

Step 2: Select the degree. Degree 2 has the lowest validation MSE, $1$, so the analyst fixes the selected degree at 2. The reported training MSEs decrease with degree, but they do not overturn the validation comparison.

Step 3: Refit and assess. The degree-2 model is now refit using the training and validation observations together. Suppose its resulting test residuals are $2,-1,0$. The test MSE is $(2^2+(-1)^2+0^2)/3=(4+1+0)/3=5/3=1.6667$ to four decimal places. This value assesses the refitted selected model on the test observations. It is not compared with test losses for the unselected degrees to make a new selection.

The refit uses more observations to estimate the already-selected degree. The test set remains separate from both selection and refitting. This method avoids using test performance as another tuning signal.

Common Mistakes and Interpretation

Choosing the smallest training loss. Training loss generally rewards flexibility for nested least-squares polynomials. Choose using the stated validation criterion, not the training column.
Forgetting to average. For residuals $1,-2,0$, the squared-error sum is $1+4+0=5$, but MSE is $5/3$, not 5. Divide by the number of evaluated observations.
Changing the loss midway. If the task specifies MAE, do not rank candidates by MSE because one is easier to calculate. Apply the stated loss consistently.
Letting the test set break a tie. A tie should be handled by a prespecified rule. If the test result changes the choice, it has become part of model selection rather than an independent final assessment.
Comparing totals across different sample sizes. If validation sets have different sizes, a total error sum can favor the smaller set. Compare average loss per observation, with appropriate care if the evaluation sets are not comparable.
Assuming the smallest observed validation loss is certain to generalize. Validation results are based on a finite sample and can vary across splits. They guide selection; the separate test set provides a final assessment, not a guarantee of future performance.

Key takeaway: State the candidate degrees, the evaluation loss, and any tie rule. Compute each candidate's validation loss correctly, select from validation results, and reserve the test set for the final assessment of the selected procedure.

Check Your Understanding

For each problem, use only the stated loss and selection information. Answers are included for checking.

A degree-2 model has validation residuals $1,-1,2$. Calculate its validation MSE. (Answer: $2$.)
Two polynomial degrees have validation residuals as follows: degree 1 has $2,0$, and degree 2 has $1,1$. Using MSE, which degree is selected? (Answer: degree 2; validation MSEs are $2$ and $1$, respectively.)
A stated MAE criterion is used. Degree 2 has validation residuals $0,-3,1$, and degree 3 has $1,1,1$. Which degree has lower validation MAE? (Answer: degree 3; its MAE is $1$, compared with $4/3$ for degree 2.)
Validation MSE is 2 for degree 1 and 1 for degree 2. After degree 2 is selected, its test residuals are $2,-2$. Calculate the final test MSE, and state whether the validation comparison should be changed. (Answer: test MSE $=4$; do not change the selected degree based on this test result.)
Validation MSEs for degrees 1, 2, and 3 are $2,1,1$, respectively. The tie rule, stated before evaluation, is to choose the lower degree. Training MSEs are $5,2,1$, respectively. Which degree is selected, and why is degree 3 not selected? (Answer: degree 2; it ties for the lowest validation MSE and is the lower-degree tied candidate. Degree 3's lower training MSE does not determine selection.)

What You'll Learn