Why Compare Transformations Carefully?
A transformation changes the response values used in an analysis of variance (ANOVA). It may make the residual spread more consistent across groups or make residuals more nearly normal. But a transformation also changes the scale of the analysis. Selecting one just because it makes an F test significant confuses two different questions: whether the model’s assumptions are reasonably supported, and whether the group means differ.
The aim here is to compare plausible transformations by their effect on residual diagnostics and by whether the resulting scale is useful. The groups, observations, and design should stay the same in each comparison. In particular, do not try many transformations and keep the one that gives the smallest p-value.
The Model and What to Compare
Suppose a study has \(k\) groups, with \(n_i\) independent observations in group \(i\). For a candidate transformation \(g\), let \(z_{ij}=g(y_{ij})\) be the transformed response for observation \(j\) in group \(i\). A one-way ANOVA model on that scale is
Here \(\mu_i\) is the mean transformed response in group \(i\), and \(\varepsilon_{ij}\) is the error for that observation. The classical ANOVA model assumes that errors are independent, normally distributed within each group, and have a common variance \(\sigma^2\). Independence must be justified by the study design; a transformation does not make dependent observations independent.
The fitted value for an observation is its group’s transformed sample mean, \(\bar{z}_i\). Its residual is the observed transformed value minus that fitted value:
Residual plots and Q-Q plots help assess whether the equal-variance and approximate-normality assumptions are plausible. Compare these diagnostics across candidate scales, looking for systematic patterns and substantial departures rather than demanding a perfect-looking plot. For a one-way ANOVA, a residual-versus-fitted plot shows residuals against the group means; a residual-versus-group display can make differences in spread easier to see.
A comparison is meaningful only when the candidates are applied to the same observations and the same groups. If one transformation requires dropping observations or changing the groups, differences in fit may reflect those changes rather than the transformation. Also check that the transformation is defined for every response: for example, \(\log(y)\) requires \(y>0\).
Worked Example: Multiplicative Spread and a Log Transformation
Worked Example: Multiplicative Spread and a Log Transformation
Consider hypothetical measurements from four independent experimental units in each of three groups. The responses are \(10,12,14,16\), \(20,24,28,32\), and \(40,48,56,64\). The measurements are positive, so both the original response and its natural logarithm can be analyzed. Assume the groups were assigned or sampled in a way that supports independence.
On the original scale, the group means are \(13\), \(26\), and \(52\). The residuals in the first group are \(-3,-1,1,3\), so its residual sum of squares is \(9+1+1+9=20\), and its sample residual variance is \(20/(4-1)=6.667\) squared measurement units. The second group’s residuals are \(-6,-2,2,6\), giving variance \(80/3=26.67\). The third group’s residuals are \(-12,-4,4,12\), giving variance \(320/3=106.7\). The spread grows markedly from group to group.
Now fit the model to \(z=\log(y)\). Because the observations in each successive group are twice those in the preceding group, the log values in each group are shifted by \(\log(2)\). Adding the same constant to every value in a group does not change its residuals. In the first group, the mean log response is approximately \(2.5498\), giving residuals \(-0.2472,-0.0649,0.0893,0.2228\). Thus its residual variance is \((0.2472^2+0.0649^2+0.0893^2+0.2228^2)/3\approx0.0410\). The other two groups have the same residuals and variance.
This pattern supports the log scale as a candidate: the large differences in residual spread on the original scale disappear on the log scale. The residual values are not a guarantee of normal errors, and four observations per group are too few to assess normality confidently. The independence assumption still depends on the design. A log transformation is defensible here because it addresses a visible spread pattern, not because of an F test.
Why the F Statistic Is Not a Selection Rule
For a one-way ANOVA, the F statistic compares variation among the fitted group means with residual variation within groups. Changing the response scale changes both quantities, so F statistics from different transformations are not a simple contest in which the larger value identifies the better model. A small p-value is evidence about group means under the model fitted on that scale; it is not evidence that the scale’s assumptions are better satisfied.
In the preceding example, the original-scale grand mean is \(30.333\). The between-group sum of squares is \(4[(13-30.333)^2+(26-30.333)^2+(52-30.333)^2]\approx3154.7\). The within-group sum of squares is \(20+80+320=420\). Therefore,
On the log scale, the group means are approximately \(2.5498\), \(3.2429\), and \(3.9361\), and the grand mean is \(3.2429\). The between-group sum of squares is approximately \(3.8436\); the within-group sum of squares is approximately \(3(0.1229)=0.3688\). Hence
These F values differ because they describe variation on different scales. That difference does not by itself show that the log model is superior. The relevant reason to favor it in this example is the improved residual-spread pattern, considered alongside normality diagnostics, design assumptions, and the usefulness of the scale.
Worked Example: Comparing Square-Root and Log Scales
Worked Example: Comparing Square-Root and Log Scales
Suppose three independent groups have hypothetical positive count responses: group A has \(4,9,16,25\), group B has \(9,16,25,36\), and group C has \(16,25,36,49\). Counts are positive, so both \(\sqrt{y}\) and \(\log(y)\) are defined. The study design supports independence; the question is which scale better addresses the observed spread.
On the original scale, the sample means are \(13.5\), \(21.5\), and \(31.5\). For group A, the residuals are \(-9.5,-4.5,2.5,11.5\), so the sample residual variance is \((90.25+20.25+6.25+132.25)/3=83.00\). The corresponding calculations give \(409/3=136.3\) for B and \(609/3=203.0\) for C. The residual spread increases across groups.
For the square-root transformation, the transformed values are \(2,3,4,5\) in A; \(3,4,5,6\) in B; and \(4,5,6,7\) in C. Each group has residuals \(-1.5,-0.5,0.5,1.5\). Thus each sample residual variance is \((2.25+0.25+0.25+2.25)/3=1.667\). Equal residual variances are more plausible under this candidate.
For comparison, the log-transformed residual variances are approximately \(0.626\), \(0.357\), and \(0.233\), respectively. They still decline across groups. The square-root scale therefore gives the clearer equal-spread pattern among these choices. This does not prove that its residuals are normal: inspect the residual Q-Q plot as well, bearing in mind that four observations per group provide limited information. The choice is based on the diagnostics together, not on whichever transformation yields the preferred test result.
A Short Justification: Why Residuals Are Centered Within Groups
In a one-way ANOVA, the fitted value for every observation in group \(i\) is \(\bar{z}_i\), the mean of that group’s transformed responses. Therefore the residuals in that group sum to zero:
The last equality follows from \(\bar{z}_i=(\sum_j z_{ij})/n_i\). This is why residual plots are useful for checking patterns around the fitted group means: the residuals describe deviations left after those means have been fitted. A group’s residual variance is the residual sum of squares divided by \(n_i-1\), when \(n_i>1\). Comparing such variances is informative, but plots and context matter too; a single summary cannot reveal every pattern.
Worked Example: When Keeping the Original Scale Is Better
Worked Example: When Keeping the Original Scale Is Better
Consider hypothetical measurements from three independent groups: A has \(0,1,1,2\), B has \(1,2,2,3\), and C has \(2,3,3,4\). The design supports independence. On the original scale, each group has residuals \(-1,0,0,1\), so each sample residual variance is \((1+0+0+1)/3=0.6667\) squared measurement units. The spreads are equal, and the residual patterns are identical across groups.
A square-root transformation is defined for these nonnegative responses, but it does not improve the spread pattern. The transformed group variances are approximately \(0.362\), \(0.0901\), and \(0.0574\), so they differ substantially. The transformation creates a pattern not evident on the original scale. A logarithm of the response is undefined for the zero; using \(\log(y+1)\) would be a different, shifted transformation whose offset needs a defensible reason.
With just four observations per group, normality remains difficult to assess, even though the residuals look symmetric. If the residual Q-Q plot and study context raise no serious concern, retaining the original scale is reasonable: the transformation has not improved the equal-spread diagnostic and would make the response less directly interpretable. This is not a claim that the original model is certainly correct; it is a decision based on the evidence available.
A Practical Comparison Strategy
Use the same groups and observations for each candidate where possible. Check that independence is supported by how the data were collected.
Confirm that every response is in the domain of each transformation. Document any shift or offset and why it is appropriate.
Inspect residual spread by group and against fitted values, and inspect a Q-Q plot for substantial nonnormality. Consider whether any improvement is clear rather than marginal.
Ask whether the transformed response is meaningful for the scientific question and whether the gain in diagnostics justifies using it.
State the chosen scale and the diagnostic evidence supporting it. Do not use the smallest p-value or largest F statistic as the selection rule.
Common Mistake
Wrong reasoning: “I tried several transformations and chose the one with \(p<0.05\), so its assumptions must be better.” This is wrong because significance tests answer a question about group means conditional on the chosen model. They do not measure how well the model assumptions hold, and searching across transformations for significance can make the result misleading.
Correction: Compare residual patterns and the plausibility of the model assumptions first. Use interpretability and the purpose of the analysis as part of the decision. Report the rationale for the chosen scale, and do not present the significance of an F test as proof that the transformation was appropriate.
Check Your Understanding
Answer each question using the diagnostic and selection principles in this tutorial.
- What three assumptions are used in the classical one-way ANOVA model for a transformed response?
- Two candidate transformations give different F statistics. Why is choosing the larger one not a sound selection rule?
- In the square-root example, each group had the same residual variance. What did that suggest, and what did it not establish?
- Why should a logarithm of a response containing zero not be applied without addressing the transformation’s domain?
- When might retaining the original response scale be preferable to using a transformation?