From a Curved Scatterplot to a Candidate Model
A transformation can make a curved relationship easier to describe with a straight-line regression. But a curve that looks plausible on a scatterplot is only a reason to try a particular transformation; it is not evidence that the resulting model fits well. Residual plots provide the next check: if a pattern remains after fitting, the chosen model has not adequately captured the relationship, or another regression condition may be questionable.
The goal is therefore not to find a transformation that makes a graph look straight at any cost. It is to propose a model that suits the shape of the data, fit it on the appropriate scale, and examine its residuals on that same scale. The data and study design still matter: a transformation does not make dependent observations independent or repair an influential observation.
For example, a model fitted to \(\ln(y)\) has residuals in log-response units, not in the original units of \(y\). A plot of those residuals against the fitted log response assesses the model on the scale on which its line was fitted. Residuals should generally fluctuate around zero without a systematic curve, and their vertical spread should be reasonably similar across the range of fitted values. Whether those conditions are plausible is judged alongside the study design and the amount of data.
Specify the Model Before Checking It
A transformed straight-line regression can be written as \(T(y_i)=\beta_0+\beta_1g(x_i)+\epsilon_i\). Here, \(T\) transforms the response (possibly leaving it unchanged), and \(g\) transforms the predictor (also possibly leaving it unchanged). The regression assumptions concern the errors \(\epsilon_i\) on this fitted scale: they have mean zero, are independent across observations, and have approximately constant variance. Approximate normality of errors is also used for standard regression inference in small samples. The design should support independence; a residual plot alone cannot establish it.
Candidate forms encountered in this unit include an untransformed line, a line using a logarithm of the response or predictor, square-root forms, and reciprocal forms. For a logarithm or reciprocal, the values being transformed must be in the transformation's domain: for example, a logarithm requires a positive value, and a reciprocal requires a nonzero value. Which variable is transformed depends on the pattern and the scientific meaning of the quantities.
| Visual clue in the original scatterplot | Candidate to try | What to inspect after fitting |
|---|---|---|
| Points follow an approximately straight band | Untransformed response and predictor | Residuals should show no remaining curve or clear change in spread |
| Response rises or falls at an increasing rate | A log-response model may suit exponential-type growth or decay | Residuals for the log response should be patternless on that scale |
| Change in response is rapid at first and then slows | A log-predictor model may suit a logarithmic pattern | Check residuals against fitted values and transformed predictor |
| Curvature resembles a square-root or reciprocal relationship | Try the relevant square-root or reciprocal form | Check for residual curvature and changing spread on the fitted scale |
These are clues, not rules. A scatterplot can be noisy, and different candidate models may look similarly plausible. The residual plot helps distinguish them. A systematic U-shape or arch suggests remaining curvature; a fan-shaped spread suggests nonconstant variability; and a single large residual can merit checking the observation and its context. Such patterns are warnings to investigate, not automatic instructions to transform.
Worked Examples: Fit, Then Diagnose
Worked Example: Exponential-Type Growth
Suppose five hypothetical, independent observations have predictor values \(x=0,1,2,3,4\) and responses \(y=3,6,12,24,48\). These values are deliberately chosen to show a clear exponential pattern. All responses are positive, so a log-response model is defined. Compare an untransformed line with a line for \(\ln(y)\) against \(x\).
For the untransformed fit, \(\bar{x}=2\) and \(\bar{y}=18.6\). The least-squares slope and intercept are:
The fitted values are \(-3,7.8,18.6,29.4,40.2\), giving residuals \(6,-1.8,-6.6,-5.4,7.8\). The positive residuals at the ends and negative residuals in the middle form a curved pattern. The straight line misses the systematic growth.
For the transformed model, the logged responses are \(\ln(3),\ln(6),\ln(12),\ln(24),\ln(48)\). Each successive value increases by \(\ln(2)\), so the fitted equation is \(\widehat{\ln(y)}=\ln(3)+\ln(2)x\). Substituting the five \(x\) values reproduces each logged response exactly; the residuals on the log scale are all zero, apart from rounding.
The log-response model matches this idealized pattern better. However, zero residuals in deliberately exact data do not let us assess residual spread or normality. In a real analysis, the independence assumption must be supported by how observations were obtained, and residual plots from data with noise must be checked. This example illustrates a candidate choice; it does not establish those assumptions for an actual study.
Worked Example: A Square-Root Response
Consider hypothetical observations \(x=1,2,3,4,5\) and \(y=4,9,16,25,36\). The responses are positive. The original scatterplot bends upward, and the square roots of the responses are \(2,3,4,5,6\), suggesting a line for \(\sqrt{y}\) against \(x\).
For comparison, the untransformed response has \(\bar{x}=3\) and \(\bar{y}=18\). Its least-squares slope is \(\sum (x_i-3)(y_i-18)/\sum (x_i-3)^2=80/10=8\), and its intercept is \(18-8(3)=-6\). The fitted values are \(2,10,18,26,34\); subtracting these from the observed responses gives residuals \(2,-1,-2,-1,2\). Their positive ends and negative middle indicate curvature left by the untransformed line.
On the square-root scale, the points satisfy \(\sqrt{y}=1+x\). The fitted line therefore has intercept \(1\) and slope \(1\); substituting each \(x\) gives fitted square roots \(2,3,4,5,6\), so all five residuals on that scale are zero. This noiseless example cannot verify constant variance or normality, and independence still depends on the hypothetical sampling design. With real observations, those conditions require assessment rather than inference from a perfect fit.
The transformation changes the response scale, so residuals from this fit describe errors in square-root units. It is not appropriate to inspect only a plot of original-scale responses and declare the transformed regression adequate; examine its residuals on the scale used for fitting.
Worked Example: A Reciprocal Predictor
Suppose a hypothetical process is observed at positive settings \(x=1,2,4,8\), with responses \(y=30,20,15,12.5\). The response decreases rapidly at small \(x\), then levels off. A reciprocal-predictor model is a reasonable candidate because \(1/x\) is defined for all four predictor values.
First fit the untransformed line. Here \(\bar{x}=3.75\), \(\bar{y}=19.375\), \(\sum(x_i-\bar{x})(y_i-\bar{y})=-60.625\), and \(\sum(x_i-\bar{x})^2=28.75\). Thus,
The fitted values are approximately \(25.1739,23.0652,18.8478,10.4130\), and the residuals are \(4.8261,-3.0652,-3.8478,2.0870\). The ends are positive and the middle values negative, again indicating that the untransformed line leaves curvature.
Now let \(z=1/x\). The predictor values become \(1,0.5,0.25,0.125\), and the responses satisfy \(y=10+20z\) exactly: for example, at \(x=4\), \(z=0.25\) and \(10+20(0.25)=15\). The fitted line has intercept \(10\) and slope \(20\); its residuals on the response scale are all zero. As in the earlier examples, exact illustrative values cannot demonstrate realistic error variation or normality. The observation design must support independence, and noisy data would need residual checks.
This is a predictor transformation: the response remains in its original units. It differs from fitting a reciprocal response, which would instead use \(1/y\) as the response. Choose between such forms by the observed pattern and subject-matter context, then check the resulting fit; do not treat “reciprocal” as a single universal model.
A Practical Selection and Checking Process
The examples show how a residual pattern can disqualify an otherwise convenient model and how a transformation can remove a visible curve. A disciplined comparison keeps the model assumptions and residual scale explicit.
Identify whether the scatterplot is approximately straight, bends increasingly, levels off, or has another systematic shape. Check units and whether candidate transformations are defined for the observed values.
Specify which variable is transformed and write the proposed straight-line relationship on that scale. State the error assumptions for that model before interpreting its residuals.
Plot residuals against fitted values and, where useful, against the predictor on the fitted scale. Look for systematic curvature, changing spread, and unusual observations.
If a clear pattern remains, reconsider the candidate. Prefer a model with an interpretable relationship and residuals without an obvious systematic pattern, provided the design and assumptions are defensible.
A short justification explains why residual plots are useful. If the model is \(T(y_i)=\beta_0+\beta_1g(x_i)+\epsilon_i\), then the fitted value is \(\widehat{T(y_i)}\), and the residual is \(e_i=T(y_i)-\widehat{T(y_i)}\). If the mean relationship is adequately represented by the line and the errors have mean zero, residuals should not consistently be positive in one region and negative in another. A curve in a residual plot is evidence that the line has missed some systematic feature of the mean relationship. If error variance is approximately constant, the residuals should also have roughly similar vertical spread across fitted values. These expectations explain why residual patterns matter; they do not prove the assumptions, especially with few observations.
Common Mistakes and Why They Matter
- Choosing the model only by R-squared. A larger R-squared does not guarantee a good model. In particular, R-squared values from models with differently transformed responses describe variation on different scales and should not be compared as if they measured the same thing. Inspect residuals and consider whether the model is meaningful in context.
- Using the wrong residual scale. For a log-response regression, \(e_i=\ln(y_i)-\widehat{\ln(y_i)}\). These are not original-response residuals. Diagnose the regression using the residuals from the scale actually fitted.
- Assuming a transformation repairs every problem. A transformation may address curvature or a variance pattern, but it cannot fix dependence caused by the study design, automatically resolve influential observations, or show that the model is scientifically appropriate.
- Treating a visual clue as a conclusion. “The scatterplot looks exponential” is a reason to try a log-response model, not a final diagnosis. If its residuals still show curvature or changing spread, the candidate needs reconsideration.
Check Your Understanding
For each question, identify the model or diagnostic reasoning that is appropriate.
- A positive response rises faster as the predictor increases. Which candidate transformation might you try, and what residual plot would help evaluate it?
- A model uses \(\ln(y)\) as its response. In what units or scale are its residuals, and why does that matter when diagnosing the fit?
- A reciprocal-predictor model has a residual plot with a clear arch. What does the pattern suggest, and what should you do next?
- Why is a higher R-squared not, by itself, sufficient reason to select one transformed model over another?
- What can residual plots reveal about a fitted model, and what important assumption must instead be supported by the study design?