Recognizing Underfitting

Introduction

A model can perform poorly on new observations because it has learned too little about the relationship in the data. This is underfitting: the fitted model is too restricted to represent important patterns, so its predictions remain inaccurate even on the observations used to fit it. The key signal is not merely a small difference between training and validation error. Both errors must also be judged against a meaningful accuracy target or a suitable alternative model.

For an actuarial prediction task, a model that misses systematic differences in claim cost, lapse probability, or expected frequency may produce poor decisions even if its validation error is close to its training error. The diagnosis therefore combines numerical performance with the structure of the model and the problem. The examples below use hypothetical data and squared-error loss.

Definition: A model is underfitting when its fitted predictions fail to capture important structure in the data, leading to unacceptably poor performance on the training data and typically also on validation data. The cause may be excessive bias, inadequate flexibility, a misspecified model form, or overly strong regularization. A small training-validation gap alone is not a definition of underfitting.

Reading the Error Pattern

Suppose the response is quantitative, the same prediction target and loss are used on training and validation observations, and lower loss means better predictions. Under squared-error loss, the mean squared error (MSE) on a set of $n$ observations is the average squared difference between each observed response and its prediction:

\mathrm{MSE}=\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2.

A common underfitting pattern is high training MSE and high validation MSE, often with the two values not far apart. That pattern suggests the model is not capturing enough of the signal even where it was fitted. “High” is contextual: compare the errors with a stated accuracy requirement, a useful baseline, or a more flexible model evaluated on the same data and by the same loss. There is no universal MSE value that establishes underfitting across different response scales and applications.

Training and validation errors that are both low do not indicate underfitting. Nor is a small difference between them sufficient evidence: two errors can be close because both are acceptable, or because both are poor. Validation results also depend on whether the validation observations represent the prediction setting; a misleading or unusually small validation set can make a diagnosis uncertain.

Diagnostic principle: Treat underfitting as a joint diagnosis. Look for poor training performance, poor validation performance, and evidence that the model cannot express relevant structure. Then check whether a reasonable change to model form or flexibility improves performance.

Why Restricted Models Cannot Improve Training Fit

For squared-error fitting, a model class is a set of candidate prediction rules. If one class is contained within another, the more flexible class can always reproduce any predictions available to the restricted class. This gives a useful result about training error.

Result: When models are fitted by minimizing the same training squared-error criterion, the minimum training MSE over a restricted class cannot be less than the minimum training MSE over a class that contains it.

To see why, let $\mathcal{M}_1$ be the restricted class and $\mathcal{M}_2$ the larger class, with $\mathcal{M}_1\subseteq\mathcal{M}_2$. Every candidate in $\mathcal{M}_1$ is also a candidate in $\mathcal{M}_2$. Therefore the smallest training error available in $\mathcal{M}_2$ is no larger than the smallest one available in $\mathcal{M}_1$:

\min_{m\in\mathcal{M}_2}\mathrm{MSE}_{\mathrm{train}}(m) \leq \min_{m\in\mathcal{M}_1}\mathrm{MSE}_{\mathrm{train}}(m).

This result explains why adding flexibility can reduce training error, but it does not guarantee lower validation error. The inequality concerns the training criterion only. For an underfitting diagnosis, the practical question is whether the restricted model’s training error is already unacceptably large and whether a justified alternative can represent the missing pattern. The result assumes both classes use the same observations and loss and are optimized over their stated candidates; it need not describe models that are fitted using different objectives or optimization procedures.

Worked Examples

Worked Example: Comparing a Restricted Model with a Flexible Alternative

A hypothetical insurer predicts a numerical renewal-cost measure. The response is measured in units of $100. A restricted model and a more flexible candidate are evaluated on the same four training observations and four validation observations. Their residuals, defined as observed response minus prediction, are:

Restricted model, training: $3,-3,2,-2$. Restricted model, validation: $2,-2,3,-3$. Flexible candidate, training: $1,-1,0,0$. Flexible candidate, validation: $1,-1,1,-1$. For this task, the stated maximum acceptable MSE is $1.2$ squared units. Diagnose the restricted model.

Method. Compute the MSE for each set by squaring each residual, summing the squares, and dividing by four. Then compare each result with the accuracy target and with the flexible candidate.

For the restricted model on training data, the squared residuals are $9,9,4,4$. Their sum is $26$, so:

\mathrm{MSE}_{\mathrm{train}}=\frac{26}{4}=6.5.

On validation data, the squared residuals are $4,4,9,9$, again summing to $26$:

\mathrm{MSE}_{\mathrm{validation}}=\frac{26}{4}=6.5.

Both errors exceed the target of $1.2$. For the flexible candidate, training squared residuals sum to $1+1+0+0=2$, giving training MSE $2/4=0.5$. Validation squared residuals sum to $1+1+1+1=4$, giving validation MSE $4/4=1.0$. Both candidate errors meet the stated target.

Conclusion. The restricted model has high training and validation MSE, and a candidate evaluated on the same observations has much lower error and meets the target. This supports diagnosing the restricted model as underfitting. The comparison is meaningful because the response units, observations, and loss are held fixed.

Common mistake avoided: Calling the restricted model adequate because its training and validation errors are equal. Their equality describes the gap, not accuracy; here both values are far above the specified target.

Worked Example: Detecting a Misspecified Functional Form

A hypothetical analyst models an outcome using a straight-line regression on a single predictor $x$, although the underlying relationship in the supplied data is $y=x^2$. The training predictor values are $-2,-1,0,1,2$. The validation values are $-1.5,-0.5,0.5,1.5$. Find the least-squares straight line in each set, calculate both MSEs, and assess whether the linear form is adequate.

Model and method. Fit an intercept and slope by ordinary least squares, minimizing the sum of squared residuals. In each set the $x$ values are symmetric about zero, so $\sum x_i=0$ and $\sum x_i y_i=0$. The fitted slope is therefore zero, and the fitted intercept is the mean of the observed $y$ values.

In training, the responses are $4,1,0,1,4$, which sum to $10$. The fitted intercept is $10/5=2$, so the fitted line predicts $2$ at every training $x$. The residuals are $2,-1,-2,-1,2$. Their squared values sum to $4+1+4+1+4=14$, giving:

\mathrm{MSE}_{\mathrm{train}}=\frac{14}{5}=2.8.

In validation, the responses are $2.25,0.25,0.25,2.25$, which sum to $5$. The fitted intercept is $5/4=1.25$, and the slope is again zero. The residuals are $1,-1,-1,1$. Their squared values sum to $4$, giving:

\mathrm{MSE}_{\mathrm{validation}}=\frac{4}{4}=1.

The straight-line model misses the curvature in both sets. A quadratic model $y=\beta_0+\beta_1x+\beta_2x^2$ with coefficients $\beta_0=0,\ \beta_1=0,\ \beta_2=1$ reproduces every supplied response, so its MSE is zero on both sets. Within this deliberately exact example, the poor linear fit is explained by a misspecified functional form rather than by random variation.

Conclusion. The linear form is inadequate for the stated relationship. The curved pattern is the important evidence: merely observing different training and validation MSEs would not identify the reason for poor fit.

Common mistake avoided: Assuming a straight line is suitable simply because there is one predictor. The number of predictors does not determine whether the chosen functional form captures the relationship.

Worked Example: Excessive Regularization Shrinking a Signal

A hypothetical analyst uses a centered predictor $x$ to predict a centered outcome $y$, and fits a regression through the origin. The training values satisfy $y=2x$ for $x=-2,-1,1,2$. The analyst minimizes the ridge objective $\sum_i(y_i-\beta x_i)^2+\lambda\beta^2$ using $\lambda=10$. Validation predictor values are $-1.5,-0.5,0.5,1.5$, with outcomes also satisfying $y=2x$. Calculate both MSEs and compare them with the unpenalized fit. Assume an MSE above $0.2$ is unacceptable for this task.

Model and method. The regression is through the origin, the data are centered, and the penalty applies to the slope $\beta$. Differentiate the stated objective with respect to $\beta$, set the derivative to zero, and solve:

-2\sum_i x_i(y_i-\beta x_i)+2\lambda\beta=0, \qquad \hat{\beta}_{\lambda}=\frac{\sum_i x_i y_i}{\sum_i x_i^2+\lambda}.

For training data, $\sum x_i^2=4+1+1+4=10$. Since $y_i=2x_i$, $\sum x_i y_i=2\sum x_i^2=20$. Thus the penalized slope is $20/(10+10)=1$. The residuals $y_i-\hat{\beta}_{\lambda}x_i$ equal $x_i$, so their squares sum to $10$. The training MSE is $10/4=2.5$.

For validation data, the penalized slope remains $1$, as the fitted model is unchanged. The residuals are the validation $x$ values, whose squares sum to $2.25+0.25+0.25+2.25=5$. Therefore the validation MSE is $5/4=1.25$. Both values exceed the $0.2$ target.

With $\lambda=0$, the slope is $20/10=2$, which reproduces the relationship $y=2x$ exactly. Training and validation residuals are then all zero, so both MSEs are zero. For this hypothetical signal, the large penalty has shrunk the slope too far and caused underfitting.

Conclusion. Excessive regularization can restrict effective model flexibility even when the underlying model form is appropriate. The calculations identify a specific remedy to investigate: reduce the penalty and reassess performance.

Common mistake avoided: Treating the penalty as if it were part of the prediction error. The ridge penalty determines the fitted slope, but the MSEs above are calculated from squared prediction residuals alone.

What the Diagnosis Does—and Does Not—Tell You

Underfitting is a description of inadequate predictive fit, not a complete explanation of why it occurred. Excessive bias is one useful interpretation: if the model class systematically cannot represent the relationship, its predictions can remain systematically displaced from the response even with a large training sample. A model may also be too simple because it omits relevant predictors, restricts the shape of an effect, or uses a penalty strong enough to suppress meaningful coefficients.

Before changing model complexity, check for other plausible causes of high training error. The response may be measured or coded incorrectly; the selected loss may not match the prediction goal; predictors may be unavailable or incorrectly constructed; or a fitting procedure may not have reached a good solution. These problems are not fixed automatically by selecting a more flexible model. Likewise, an apparently poor validation result may reflect a validation set that does not represent the intended prediction population.

Confirm the performance measure.
Check that training and validation errors use the same response scale, loss, and target. Establish what level of error is practically acceptable or select a meaningful reference.

Check both error levels.
If training and validation performance are both unacceptably poor, the model may not capture the signal. Do not use the gap alone to decide.

Inspect the model’s restrictions.
Ask whether its predictors and functional form can represent the patterns relevant to the task, and whether regularization or other constraints are too strong.

Test a justified alternative.
Compare a less restricted or differently specified candidate using the same performance measure. An improvement on both training and validation data supports the underfitting diagnosis.

The central practical distinction is between model inadequacy and a mere difference in error values. A model with both errors above a meaningful tolerance, systematic missed structure, and a sound alternative that improves fit is a strong underfitting candidate. High errors alone do not prove that adding complexity is the right solution; the model, data, optimization, and evaluation target all need to make sense together.

Key takeaway: Diagnose underfitting when a model is too restricted to capture important structure and therefore has poor training as well as poor validation performance. Use an accuracy target or a comparable alternative to establish that the errors are genuinely poor; a small training-validation gap by itself is not enough.

Check Your Understanding

For each problem, use the stated loss and comparison to decide what the evidence supports.

A model for a hypothetical benefit amount has training MSE $4.8$ and validation MSE $5.1$. The task’s acceptable MSE is $1.0$. A more flexible candidate has training MSE $0.8$ and validation MSE $0.9$ on the same observations. What is the best diagnosis? (Answer: The first model is underfitting; both of its errors exceed the target, while the comparable candidate meets it.)
A restricted model has training MSE $0.4$ and validation MSE $0.5$, with an acceptable MSE of $1.0$. Its errors differ by only $0.1$. Does the small gap show underfitting? (Answer: No. Both errors are below the stated target, so the small gap alone does not indicate underfitting.)
A model predicts $y$ from $x$ with a straight line. A plot of residuals against $x$ shows a repeated U-shaped pattern, and both training and validation MSE exceed the application’s tolerance. Name a plausible model-form issue. (Answer: The straight-line form may be misspecified because it fails to represent curvature.)
For a centered through-origin ridge fit, $\sum x_i^2=8$, $\sum x_i y_i=24$, and $\lambda=4$. Calculate the fitted slope. If the penalty is known to suppress a real positive signal and both errors are poor, what cause of underfitting is plausible? (Answer: The slope is $24/(8+4)=2$; an overly strong penalty is a plausible cause.)
A candidate has high training and validation MSE, but the outcome variable was accidentally recorded in cents for training and dollars for validation. Is underfitting established, and what should be checked first? (Answer: No. The errors are not comparable; correct the inconsistent outcome scales before diagnosing underfitting.)

What You'll Learn