Choosing a Transformation Is a Modeling Decision
A scatterplot can show two problems at once: the average response may bend with the predictor, while the vertical spread of observations grows as the fitted response increases. These features call for different reasoning. A transformation of the predictor changes the shape of the fitted mean pattern; a transformation of the response changes the scale on which deviations are measured. Choosing one transformation simply because it makes a plot look straighter can therefore leave a major problem unresolved.
A careful comparison asks whether a candidate model captures the systematic pattern, whether its residuals have a suitable shape and spread, and whether its scale-specific interpretation is useful. The examples use hypothetical data and fitted summaries. The goal is not to find a transformation that makes every diagnostic perfect, but to justify a model that is defensible for the question being asked.
State the Model Before Comparing Fits
For ordinary least squares regression of a response \(Y\) on predictors \(Z_1,\ldots,Z_p\), write \(Y_i=\beta_0+\sum_{j=1}^p\beta_j Z_{ij}+\varepsilon_i\). The usual conditions for the standard coefficient tests and confidence intervals are that the errors have mean zero conditional on the predictors, are uncorrelated across observations, and have constant variance; normal errors are additionally used for exact finite-sample \(t\)- and \(F\)-inference. These conditions concern the model as specified, not the untransformed variables automatically.
A predictor transformation changes the fitted conditional mean pattern while retaining \(Y\) as the response. For example, fitting \(Y=\beta_0+\beta_1\log X+\varepsilon\) permits a curved pattern in \(X\), but its residuals are still measured in units of \(Y\). A response transformation changes both the modeled pattern and the units of residual variation. In a log-response model, \(\log Y=\beta_0+\beta_1 X+\varepsilon\), the linearity and constant-variance assumptions apply to \(\log Y\), not directly to \(Y\). A log response requires strictly positive observations; a reciprocal response also cannot be used when an observation is zero.
To compare candidate transformations, make residual plots against fitted values and relevant predictors on the scale on which each model was fitted. Check for remaining curvature, changing spread, and unusual observations. When two models use the same response and observations, their residual sums of squares (SSE) can be compared directly. When their responses differ, such as \(Y\) versus \(\log Y\), the raw SSEs have different units and are not a direct measure of which model is better.
Worked Example: Does a Quadratic Term Capture the Curvature?
A hypothetical analyst studies annual claim counts \(Y\) against a centered exposure index \(X\). A plot suggests curvature, but the spread appears roughly constant in the observed range. For the same 10 observations, an orthogonal-polynomial calculation gives a centered response sum of squares of 18,040. The linear basis column \(q_1\) has \(\sum q_{1i}^2=10\) and \(\sum q_{1i}(Y_i-\bar Y)=386\). A quadratic basis column \(q_2\), orthogonal to the intercept and \(q_1\), has \(\sum q_{2i}^2=14\) and \(\sum q_{2i}(Y_i-\bar Y)=194\). Compare the linear and quadratic fits on the original response scale.
Method. Because the basis columns are orthogonal, the amount of centered response variation explained by each column is its squared cross-product with the centered response divided by its sum of squares. The intercept accounts for the response mean; the centered response sum of squares, 18,040, is the total variation left to explain after fitting the intercept.
Linear model. The linear component explains \(386^2/10=148,996/10=14,899.6\). Thus its residual sum of squares is \(18,040-14,899.6=3,140.4\).
Quadratic model. The additional quadratic component explains \(194^2/14=37,636/14=2,688.2857\). Therefore the quadratic residual sum of squares is \(3,140.4-2,688.2857=452.1143\), or 452.11 to two decimal places. Equivalently, \(18,040-386^2/10-194^2/14=452.1143\).
Conclusion and check. The quadratic term reduces original-scale SSE by 2,688.2857, and the remaining SSE is nonnegative and smaller than the linear-model SSE, as it must be for nested least-squares models fitted to the same observations. If residual plots also show less curvature without introducing an undesirable pattern, the quadratic fit is better supported. The calculation alone does not establish that its assumptions hold.
Common mistake. Rounding \(452.1143\) to 452.12 is incorrect: to two decimal places it is 452.11. Another mistake is to claim that the reduction in SSE proves the quadratic model is useful; adding a term to a nested least-squares model cannot increase SSE, so diagnostic evidence and the modeling purpose still matter.
Separate Mean Shape from Spread
A curved mean pattern with roughly stable vertical spread often motivates a predictor transformation or a polynomial term. For example, a log predictor gives a model linear in \(\log X\), while a quadratic model adds \(X^2\). These are not interchangeable choices: their shapes differ, and their coefficients answer different questions. A response transformation is more relevant when the spread or error behavior depends on the response scale as well as when it changes the mean shape.
The distinction is especially important when spread increases with the fitted response. If errors are approximately proportional to the response level, dividing by the level suggests that relative deviations may be more stable than absolute deviations. For positive \(Y\), taking logarithms turns multiplicative differences into additive ones: \(\log(Y/\mu)=\log Y-\log\mu\). Thus a log response can make a multiplicative pattern easier to represent and may stabilize spread. This is a modeling rationale, not a guarantee; the fitted log-scale residuals must still be checked.
The following derivation explains the link between a multiplicative model and a linear model on the log scale. Suppose \(Y_i=\exp(\eta_i)\exp(\varepsilon_i)\), with \(Y_i>0\), and suppose \(\eta_i\) is a linear predictor such as \(\beta_0+\beta_1 X_i\). Taking logarithms of both sides gives \(\log Y_i=\eta_i+\varepsilon_i\). If the errors have constant variance on this log scale, the deviations in \(\log Y\) do not have to grow with the level of \(Y\). On the original scale, however, the deviations are multiplied by \(\exp(\eta_i)\). This is why the log transformation can address increasing absolute spread when the underlying variation is approximately multiplicative.
The interpretation also changes. In the log-response model, a one-unit increase in \(X\) changes the fitted log response by \(\beta_1\); equivalently, it multiplies the fitted geometric scale by \(\exp(\beta_1)\). If the log errors are normal with variance \(\sigma^2\), then \(\exp(\eta_i)\) is the conditional median of \(Y_i\), while the conditional mean is \(\exp(\eta_i+\sigma^2/2)\). The distinction matters when the target is an arithmetic mean. A predictor transformation, by contrast, leaves the response scale unchanged.
Worked Example: Comparing a Quadratic Fit with a Log-Log Fit
A hypothetical engineer records a positive performance measure \(Y\) at four input levels \(X\). The observations are \((1,4),(2,16),(3,36),(4,64)\). Compare an original-scale quadratic model with a log-log model, and explain why the exact fit does not by itself decide between them.
Quadratic model. The observed values satisfy \(Y=4X^2\): at \(X=1\), \(4(1)^2=4\); at \(X=2\), \(4(2)^2=16\); at \(X=3\), \(4(3)^2=36\); and at \(X=4\), \(4(4)^2=64\). Thus the model \(Y=\beta_0+\beta_1X+\beta_2X^2+\varepsilon\) fits with \(\beta_0=0\), \(\beta_1=0\), and \(\beta_2=4\), giving zero residuals for these four points.
Log-log model. Taking natural logarithms gives \(\log Y=\log 4+2\log X\). For example, \(\log 4=1.3863\), \(\log 16=2.7726\), \(\log 36=3.5835\), and \(\log 64=4.1589\). The predictor values are \(\log 1=0\), \(\log 2=0.6931\), \(\log 3=1.0986\), and \(\log 4=1.3863\). Each pair satisfies \(\log Y=1.3863+2\log X\), to the displayed precision. The exact coefficients are intercept \(\log 4\) and slope 2, so this model also has zero residuals.
Decision. The quadratic specification describes a response proportional to the square of the original input. The log-log specification describes a constant elasticity of 2: multiplying \(X\) by a factor \(c\) multiplies the fitted response by \(c^2\). Since these data are exact and both models fit, the residual sums of squares cannot distinguish the models. The choice must draw on the mechanism, the intended interpretation, the range over which predictions will be made, and diagnostics with additional observations.
Common mistake. It would be invalid to compare the two zero SSE values as evidence that the models are equivalent. One is measured in squared units of \(Y\), the other in squared units of \(\log Y\); more importantly, the fitted relationships and their interpretations differ. With nonzero residuals, the numerical SSEs are still not directly comparable across these response scales.
Use Residual Spread to Evaluate a Response Transformation
A visual impression of increasing spread should be made more precise. Ask whether the size of deviations grows roughly with the fitted level, whether a different scale makes their size more stable, and whether that scale gives a reasonable interpretation. Do not transform the response solely to make a histogram look more symmetric: regression assumptions concern conditional residuals, and the relevant plots should be examined against fitted values and predictors.
Worked Example: Checking a Log Scale for Proportional Spread
A hypothetical monitoring study records positive readings in three groups with fitted levels 10, 20, and 40. In each group the two readings are 80% and 120% of that level: \((8,12)\), \((16,24)\), and \((32,48)\). Compare the within-group spread on the original and log scales. Treat each pair as equally weighted and calculate the variance using divisor 2.
Original scale. The group means are \((8+12)/2=10\), \((16+24)/2=20\), and \((32+48)/2=40\). Residuals around those means are \((-2,2)\), \((-4,4)\), and \((-8,8)\). The corresponding variances are \((4+4)/2=4\), \((16+16)/2=16\), and \((64+64)/2=64\). The variance quadruples each time the fitted level doubles, so absolute spread is not stable.
Log scale. Within each group, the log observations differ from their group-average log value by \(\log(0.8/\sqrt{0.96})\) and \(\log(1.2/\sqrt{0.96})\). Since \(0.8(1.2)=0.96\), these residuals are equal in magnitude and opposite in sign. Their magnitude is \(\tfrac12\log(1.2/0.8)=\tfrac12\log(1.5)=0.2027\). Thus each pair has log-scale variance \(0.2027^2\), approximately \(0.0411\), using divisor 2. The spread is the same in all three groups on this scale.
Conclusion and check. These values are consistent with proportional rather than constant absolute spread, so a log-response model is a reasonable candidate. The arithmetic means on the original scale are 10, 20, and 40, whereas the average log values correspond to geometric means of \(\sqrt{96}\), \(\sqrt{384}\), and \(\sqrt{1,536}\), respectively. That distinction warns against treating a fitted log-scale average as an original-scale arithmetic mean.
Common mistake. Saying “the log residuals are 20%” confuses scales. The log residual magnitude here is about 0.2027, a dimensionless difference in logarithms; it is the original observations that are 20% above or below their respective arithmetic group means.
A Practical Comparison of Candidate Models
A defensible choice is usually made in stages. First identify whether curvature, changing spread, or both are visible. Then fit candidates that address the diagnosed feature while preserving a scale that is meaningful for the problem. Check the residual plots for each candidate and record what the model says about the response. A transformation that improves one feature but produces an implausible pattern or an unusable interpretation may not be the best choice.
Look for curvature in the conditional mean and for spread that changes with fitted values. Do not infer that one transformation must solve both.
For curvature, consider a scientifically plausible predictor transformation or polynomial term. For positive responses with proportional variation, consider a log response. A response transformation requires a positive response for the logarithm.
Inspect residuals against fitted values and predictors. Look for remaining structure or a systematic spread pattern; do not rely on a single numerical fit statistic.
State whether coefficients describe additive changes, changes per transformed predictor unit, or multiplicative effects. If the response was transformed, say what original-scale quantity is being targeted.
A useful comparison table keeps the modeling consequences visible. It also prevents the common habit of treating every transformation as a generic “curve fixer.”
| Candidate | What it can change | What to inspect | Key caution |
|---|---|---|---|
| Polynomial terms in \(X\) | Curvature of the mean in \(X\), with \(Y\) unchanged | Residual curvature and spread on the original response scale | A smaller in-sample SSE is expected when adding terms; check whether the added shape is useful |
| \(\log X\) as predictor | Mean shape as \(X\) changes; response remains on its original scale | Residual structure and whether a change in \(\log X\) is meaningful | Requires \(X>0\); the coefficient is not a constant change per original \(X\) unit |
| \(\log Y\) as response | Mean relationship and residual scale | Log-scale residual curvature and variance | Requires \(Y>0\); back-transformed fitted values do not automatically equal conditional arithmetic means |
| \(\log Y\) on \(\log X\) | Models a power relationship and multiplicative response variation | Log-scale residual structure and suitability of a constant-elasticity interpretation | Both variables must be positive; its SSE is not directly comparable with an original-response SSE |
Common Traps and Final Checks
- Comparing SSEs from different response scales. An SSE for \(Y\) is in squared units of \(Y\); one for \(\log Y\) is in squared log units. Do not select a model by declaring whichever raw number is smaller the better fit.
- Using a predictor transform to solve a variance problem. A new predictor shape may remove curvature but does not, by itself, stabilize residual spread.
- Using a response transform to solve only a shape problem. A log response can change the mean relationship too. Check whether the transformed-scale pattern is appropriate, rather than assuming it is.
- Confusing a better fitted pattern with a valid model. Residual structure, independence, the relevant variance condition, influential observations, and the intended prediction target still matter.
- Ignoring the domain. A log response requires positive observations. A transformation that is undefined for observed or future values is not a usable model without a defensible alternative treatment.
Check Your Understanding
For each problem, identify the appropriate comparison and account for the scale being modeled.
- A hypothetical response is recorded at \(X=1,2,3,4\), giving \(Y=3,7,13,21\). Verify a quadratic relationship of the form \(Y=\beta_0+\beta_1X+\beta_2X^2\), and state whether a predictor transformation is necessary to represent these four values exactly. (Answer: \(Y=1+X+X^2\); a quadratic term represents them exactly, so a separate predictor transformation is not necessary for these observations.)
- Positive measurements are \((X,Y)=(2,12),(4,24),(8,48)\). Find the log-log slope and intercept, and describe the fitted power relationship. (Answer: \(\log Y=\log 6+\log X\), so the slope is 1, the intercept is \(\log 6\), and the fitted relationship is \(Y=6X\).)
- At fitted levels 5, 10, and 20, observations are respectively \((3.5,6.5),(7,13),(14,26)\). The two values in each group are 70% and 130% of the level. Find the magnitude of each residual from the group-average log value. (Answer: \(\tfrac12\log(1.3/0.7)\approx0.3095\) on the log scale, the same in all three groups.)
- A linear model for \(Y\) has original-scale SSE 45, while a model for \(\log Y\) has log-scale SSE 1.8, using the same positive observations. Can the SSE values alone establish which model fits better, and what should be checked next? (Answer: No; the SSEs have different units. Check residual patterns on each model’s own scale and assess the modeling objective and interpretation.)