Why Read the Shape Before Transforming?
A transformation changes the scale on which a regression relationship is modeled. Choosing one because a plot “looks curved” or because residuals “get wider” can solve the wrong problem: curvature concerns the conditional mean, while changing spread concerns the conditional variance. A useful diagnosis starts by asking which feature of the data is changing and in which direction.
In this tutorial, “response” means the outcome being modeled, and “predictor” means an input variable. The goal is not to name a transformation from a single visual clue. It is to decide whether the evidence points toward changing the horizontal scale, the vertical scale, both, or neither, and then to check the fitted model again.
Reading the Two Diagnostic Views
A scatterplot of response against predictor shows the observed relationship in the variables’ original units. Look for the broad shape of the response center: does it follow an approximately straight trend, or does it bend systematically? Also look at the vertical spread around that trend. A fan or funnel, for example, suggests that the spread may grow or shrink across the predictor range.
A residual-versus-fitted plot makes a related distinction after a model has been fitted. Let \(e_i=Y_i-\hat Y_i\) be the residual for observation \(i\), and let \(\hat Y_i\) be its fitted value. If a straight-line conditional mean is appropriate, residuals should fluctuate around zero without a systematic pattern in their centers. If the residual centers trace a curve, the fitted mean may be misspecified. If their centers stay near zero but their vertical spread changes, nonconstant variance is a concern.
| Pattern in a residual-versus-fitted plot | Primary concern | Initial question |
|---|---|---|
| Curved sequence of residual centers | Conditional mean is not adequately represented | Could changing a predictor’s scale make the mean relationship more nearly linear? |
| Centers near zero but spread widens or narrows | Conditional variance may change | Could changing the response scale make the spread more nearly constant? |
| Curved centers and changing spread | Both features may be present | Would one change address both, or are separate changes needed? |
| No clear pattern | No obvious departure in these plots | Is the visual evidence adequate, and are other assumptions plausible? |
These are diagnostic clues, not automatic rules. A residual plot can suggest a pattern, but does not establish its cause. In particular, a plot with one observation at each fitted value may reveal a systematic change in the residual centers, but it generally gives limited information about the variance at each value. Replicated observations or a broader view of the data may be needed to distinguish changing spread reliably.
A Least-Squares Check on Residual Patterns
Before interpreting a proposed residual sequence, check that it could actually come from the stated least-squares fit. For ordinary least squares with an intercept, the residuals sum to zero. More generally, the residual vector is orthogonal to every fitted value. These identities can rule out impossible examples and help distinguish a curve from a simple upward or downward drift.
The first identity follows because the intercept’s least-squares normal equation sets the residual sum to zero. For the second, write the fitted vector as \(\hat{\boldsymbol Y}=\boldsymbol X\hat{\boldsymbol\beta}\), where \(\boldsymbol X\) is the design matrix. The least-squares normal equations give \(\boldsymbol X^\mathsf{T}\boldsymbol e=\boldsymbol 0\), so
Thus a sequence of residuals that increases steadily with fitted value cannot generally be an ordinary least-squares residual sequence: it may violate the second identity. A positive-negative-positive or negative-positive-negative pattern can, by contrast, indicate curvature while satisfying both identities.
Worked Example: A Curved Residual Pattern That Passes the Check
A simple linear regression with an intercept has four equally spaced fitted values, \(1,2,3,4\), and residuals \(2,-2,-2,2\), in that order. Determine whether the residuals are compatible with least squares and what pattern they suggest. Use the method of checking the least-squares identities before interpreting the plot.
Step 1: Check the residual sum. The sum is \(2+(-2)+(-2)+2=0\), as required for a fit with an intercept.
Step 2: Check orthogonality to the fitted values. The inner product is \(1(2)+2(-2)+3(-2)+4(2)=2-4-6+8=0\). The sequence passes both checks.
Step 3: Interpret the pattern. Residuals are positive at the two ends and negative in the middle. Their centers therefore bend relative to zero rather than simply moving from negative to positive. This pattern is suggestive of curvature in the conditional mean. It is not, by itself, evidence that the variance changes.
Check. The residual sum and fitted-value inner product are both zero, so the sequence is compatible with the stated least-squares identities. The common mistake avoided here is using a steadily increasing residual sequence as a curvature example without checking whether it could be an OLS residual sequence.
Choosing Which Variable to Re-Express
The main diagnostic question is whether the problem lies in the shape of the mean, the spread around the mean, or both. Re-expressing a predictor changes the horizontal scale and can make a curved mean relationship more nearly linear. Re-expressing the response changes the vertical scale; it can alter both the apparent mean shape and the spread. A response change is therefore not a variance-only adjustment, and a predictor change does not automatically stabilize response variance.
- Curved mean, roughly steady spread: investigate whether a different scale for a predictor makes the conditional mean more nearly straight. Keep the response scale if there is no reason to change it.
- Mean centers near zero, spread changes: consider whether a response-scale change is appropriate for the way variability relates to the response level. Check that the transformed model still has a sensible mean relationship.
- Curved mean and changing spread: investigate both features. One response transformation may affect both, but verify each separately; otherwise, a predictor re-expression and a response re-expression may be needed.
- Neither feature is clear: do not transform just to make a plot look different. Consider the available evidence and the purpose of the model.
The choice should also respect the data’s meaning. A response transformation may change the interpretation of predictions, while a predictor transformation changes how the fitted relationship is expressed in terms of the original predictor. These are modeling choices, not purely cosmetic operations.
Worked Example: Separating a Fan in Spread From a Curved Mean
In a hypothetical portfolio study, a fitted model produces two residuals at each of four fitted values. At fitted value 10 the residuals are \(1,-1\); at 20 they are \(1,-1\); at 30 they are \(3,-3\); and at 40 they are \(3,-3\). Identify the main diagnostic signal and the transformation direction to investigate. Use group centers and spreads, and check least-squares compatibility.
Step 1: Compute the residual center at each fitted value. Each pair sums to zero, so each pair has mean residual zero: \((1-1)/2=0\), \((1-1)/2=0\), \((3-3)/2=0\), and \((3-3)/2=0\). There is no change in the displayed residual centers.
Step 2: Compute the within-pair sample variances. For the first two pairs, the mean is zero and the sum of squared deviations is \(1^2+(-1)^2=2\). Dividing by \(2-1\) gives sample variance 2 for each. For the last two pairs, the sum of squared deviations is \(3^2+(-3)^2=18\), giving sample variance 18 for each. The sample variance is nine times as large at fitted values 30 and 40 as at 10 and 20.
Step 3: Check compatibility with least squares. The total residual sum is zero. The fitted-value inner product is \(10(1-1)+20(1-1)+30(3-3)+40(3-3)=0\). The grouped residual pattern passes both identities.
Conclusion. The centers remain at zero while the spread increases, so changing variance is the primary concern shown here. A response-scale change may be worth investigating if it is appropriate for the data-generating context; changing only the predictor scale would not, by itself, address the displayed widening spread. The common mistake avoided is calling any increase in the size of residuals “curvature.” These small groups illustrate the calculation, but in an actual analysis two observations per group would give imprecise variance estimates.
Worked Example: A Predictor Scale That Straightens a Mean Pattern
A hypothetical equipment study records five predictor values \(x=1,4,9,16,25\) and responses \(Y=5,8,11,14,17\). The scatterplot bends upward with \(x\), while the vertical deviations around the bend appear similar in size. Compare a straight-line fit using \(x\) with a fit using \(z=\sqrt{x}\). Decide which scale addresses the mean pattern. The data are illustrative; they do not establish a general variance assumption.
Step 1: Fit the line using the original predictor. The means are \(\bar x=(1+4+9+16+25)/5=11\) and \(\bar Y=(5+8+11+14+17)/5=11\). The sum of squared deviations in \(x\) is \(100+49+4+25+196=374\). The sum of cross-products is \((-10)(-6)+(-7)(-3)+(-2)(0)+(5)(3)+(14)(6)=180\). Thus the slope is \(180/374=90/187\), and the intercept is \(11-(90/187)(11)=1067/187\).
The fitted values, in order, are \(1157/187,\ 1427/187,\ 1877/187,\ 2507/187,\ 3317/187\). Subtracting each fitted value from its response gives residuals \(-222/187,\ 69/187,\ 180/187,\ 111/187,\ -138/187\), approximately \(-1.187,\ 0.369,\ 0.963,\ 0.594,\ -0.738\). Their centers are negative at the ends and positive in the middle, a curved pattern rather than a simple monotone drift.
Step 2: Re-express the predictor. With \(z=\sqrt{x}\), the predictor values are \(1,2,3,4,5\). The responses satisfy \(Y=2+3z\) at every observation: for example, \(2+3(1)=5\) and \(2+3(5)=17\). The fitted line in \(z\) therefore has intercept 2, slope 3, and residuals all equal to zero.
Conclusion. In this constructed example, changing the predictor scale straightens the mean relationship without changing the response scale. The example has no residual variation after re-expression, so it cannot demonstrate that the variance is constant in a broader population. The common mistake avoided is transforming the response merely because the original scatterplot is curved.
Common Traps and a Practical Sequence
A visual pattern should lead to a specific question, not an automatic transformation. In particular, a residual plot can be misleading if its center and spread are treated as the same feature. A useful process is to describe them separately, choose a candidate change that targets the observed feature, and then inspect the new diagnostics.
In the scatterplot or residual plot, ask whether the mean trend or residual centers bend systematically around a straight-line fit.
Separately ask whether the vertical variation around the trend is roughly stable, widens, narrows, or changes in another systematic way.
A predictor re-expression targets the horizontal description of the mean; a response re-expression changes the vertical description and may affect both mean and spread.
Check the new scatterplot and residual-versus-fitted pattern. A change is useful only if it improves the feature it was meant to address without creating an unacceptable new problem.
- Trap: treating a fan as curvature. If residual centers remain near zero but their spread changes, do not describe that alone as a curved mean.
- Trap: treating any bend as changing variance. A systematic sequence of residual centers can signal a mean problem even when the spread is fairly stable.
- Trap: assuming a transformation fixes every assumption. A change that improves one feature must be checked against the others.
- Trap: ignoring least-squares constraints. For an OLS model with an intercept, proposed residuals must sum to zero and be orthogonal to the fitted values.
This distinction is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus: transformation decisions depend on diagnosing which feature of the regression model is inadequate, not on applying a fixed rule to a plot.
Check Your Understanding
For each problem, separate evidence about residual centers from evidence about spread before choosing a direction to investigate.
- A fitted line has four equally spaced fitted values \(1,2,3,4\) and residuals \(1,-1,-1,1\). Check the residual sum and fitted-value inner product. What pattern is suggested? (Answer: The sum is 0 and the inner product is \(1-2-3+4=0\); positive residuals at both ends and negative residuals in the middle suggest curvature.)
- A residual plot has centers near zero across its range. The residual standard deviations in two low-fitted-value groups are 2 and 2, and in two high-fitted-value groups are 6 and 6. Which feature changes, and which scale is a reasonable direction to investigate? (Answer: Spread changes while centers remain near zero; investigate whether a response re-expression is appropriate, then reassess the mean.)
- A scatterplot’s mean trend bends, but its vertical spread appears similar across the predictor range. Should the first candidate change target the predictor or response scale, and why? (Answer: Investigate the predictor scale first, because the main evidence concerns curvature in the mean; similar-looking spread alone does not call for changing the response scale.)
- For fitted values \(10,20,30\), residuals are respectively \((1,-1)\), \((2,-2)\), and \((4,-4)\). Compute the group means and sample variances, and state the primary signal. (Answer: Group means are all 0; sample variances are 2, 8, and 32, respectively, so the primary signal is increasing spread rather than changing residual centers.)
- A model shows both a U-shaped sequence of residual centers and a fan-shaped spread. What should be checked after a candidate response transformation? (Answer: Check separately whether the residual centers are now approximately patternless and whether the spread is more stable; if curvature remains, investigate the predictor scale as well.)