Why Transform the Response?
In a linear regression model, the usual constant-variance assumption concerns the response around its conditional mean: the spread of \(Y\) given predictors should not systematically increase or decrease with that mean. A residual plot that fans outward as fitted values rise can indicate that this assumption is implausible. If the mean relationship is otherwise adequately represented, transforming the response may make the spread more nearly constant and make linear-model inference more appropriate.
The choice of transformation should be motivated by the relationship between the conditional variance and conditional mean, not by a mechanical rule that every fan-shaped plot calls for a logarithm. A log transformation is natural when the standard deviation is roughly proportional to the mean; a square-root transformation is associated with variance roughly proportional to the mean; and a reciprocal transformation can be considered when variance grows roughly as the fourth power of the mean. These are patterns and approximations, not guarantees.
The Variance-Stabilizing Rule
Let \(\mu(x)=E[Y\mid X=x]\) and \(\sigma^2(x)=\operatorname{Var}(Y\mid X=x)\). Suppose \(h\) is differentiable near \(\mu(x)\). A first-order Taylor approximation around the conditional mean gives
The first term is constant conditional on \(X=x\), so it contributes no conditional variance. The second term is a constant multiple of \(Y-\mu\). Therefore, when the approximation is adequate,
To make this approximately constant, choose \(h\) so that \(|h'(\mu)|\sigma(x)\) is roughly constant. This gives a practical way to connect a variance pattern to a transformation.
Relate the conditional standard deviation \(\sigma\), or variance \(\sigma^2\), to the conditional mean \(\mu\).
Use \(|h'(\mu)|\) proportional to \(1/\sigma\), then identify a simple transformation with that derivative.
Fit the transformed-response model and inspect whether residual spread still changes with fitted values. A transformation does not repair a misspecified mean relationship by itself.
Common Variance Patterns
The following matches follow from the approximation above. They are useful starting points when the response is positive or nonnegative and the observed spread grows with the mean.
| Approximate original-scale variance pattern | Derivative needed for stabilization | Candidate response transformation |
|---|---|---|
| \(\sigma^2\) proportional to \(\mu\) | \(h'(\mu)\) proportional to \(1/\sqrt{\mu}\) | \(\sqrt{Y}\) |
| \(\sigma^2\) proportional to \(\mu^2\) | \(h'(\mu)\) proportional to \(1/\mu\) | \(\log Y\) |
| \(\sigma^2\) proportional to \(\mu^4\) | \(h'(\mu)\) proportional to \(1/\mu^2\) | \(1/Y\), or \(-1/Y\) |
For example, if \(\sigma^2=c\mu\) for a positive constant \(c\), take \(h(y)=\sqrt{y}\). Its derivative is \(1/(2\sqrt{y})\), so the approximate transformed variance is \((1/(4\mu))c\mu=c/4\). If instead \(\sigma^2=c\mu^2\), the derivative of \(\log y\) is \(1/y\), giving approximate variance \(c\). If \(\sigma^2=c\mu^4\), the derivative of \(1/y\) is \(-1/y^2\), giving approximate variance \(c\). The negative sign does not affect a variance because the derivative is squared.
The Taylor approximation is most reliable when responses are concentrated sufficiently close to their conditional means and the transformation is smooth across the relevant response range. It can be poor when the response has substantial relative variability, is close to a boundary where the transformation is steep, or has a distribution with strong skewness. The formulas motivate a candidate; residual diagnostics and the model context still matter.
Worked Examples
Worked Example: Poisson Counts and the Square-Root Transformation
A hypothetical monitoring system records counts \(Y\) per fixed observation period. At a low exposure level, assume \(Y\mid X=x\sim\operatorname{Poisson}(16)\); at a higher level, assume \(Y\mid X=x\sim\operatorname{Poisson}(64)\). For a Poisson response, the conditional mean and variance are both the parameter. Compare the approximate variance after taking square roots.
Method. Since \(\operatorname{Var}(Y\mid X=x)=\mu\), the variance pattern is proportional to the mean, suggesting \(h(Y)=\sqrt{Y}\). The derivative is \(h'(\mu)=1/(2\sqrt{\mu})\). The delta-method approximation is
At the low level, \(\mu=16\), so the original variance is \(16\), the derivative is \(1/(2\cdot4)=1/8\), and the approximate transformed variance is \((1/8)^2(16)=16/64=0.25\).
At the higher level, \(\mu=64\), so the original variance is \(64\), the derivative is \(1/(2\cdot8)=1/16\), and the approximate transformed variance is \((1/16)^2(64)=64/256=0.25\). Thus the approximation gives the same transformed variance at both levels, even though the original variance rises from \(16\) to \(64\).
Check. Each original variance equals its Poisson mean, and the derivative-squared factor decreases as the mean rises. The result is approximate; the exact variance of \(\sqrt{Y}\) for a Poisson variable is not exactly \(1/4\). The common mistake is to treat the delta-method result as an exact identity for every count distribution.
Worked Example: A Log Response and Original-Scale Predictions
For a hypothetical positive response \(Y\), suppose that at a specified predictor value, \(\log Y\mid X=x\sim N(\log 80,0.09)\), where the second normal parameter is the variance. Find the fitted median and the conditional mean of \(Y\), and explain what is stabilized.
Method. The log response has conditional variance \(0.09\), independent of its conditional mean on the log scale. If \(Z=\log Y\), then \(Z\sim N(\eta,\tau^2)\), where \(\eta=\log 80\) and \(\tau^2=0.09\). The inverse transformation of the fitted log-scale mean is \(\exp(\eta)=80\). For a normal \(Z\), \(\exp(\eta)\) is the median of \(Y=\exp(Z)\), not its arithmetic mean.
The lognormal mean formula follows by completing the square in the normal density, or equivalently from the normal moment-generating function:
Substitute the specified values: \(\tau^2/2=0.09/2=0.045\), so \(E[Y\mid X=x]=80\exp(0.045)=80(1.0460279\ldots)=83.6822\ldots\). Rounded to the nearest cent, the conditional mean is \(\$83.68\), while the fitted median is \(\$80.00\).
Also, a lognormal variable has \(\operatorname{Var}(Y)/E[Y]^2=\exp(\tau^2)-1\). Here that squared coefficient of variation is \(\exp(0.09)-1=0.0942\) to four decimal places, which does not depend on \(\eta\). This is the exact multiplicative-spread property of this specified lognormal model.
Check. The log-scale variance is \(0.09\), and the mean exceeds the median because the lognormal distribution is right-skewed. The common mistake is to report \(80\) as the original-scale arithmetic mean solely because it is the inverse of the fitted log-scale mean.
Worked Example: Reciprocal Transformation for a Fourth-Power Variance Pattern
A hypothetical positive response has two equally likely outcomes conditional on each predictor level. At one level, let \(Y\) be \(0.095\) or \(0.105\), each with probability \(1/2\); at another, let it be \(0.18\) or \(0.22\), each with probability \(1/2\). Calculate the conditional means and variances, then compare the exact variances of \(1/Y\) with the reciprocal delta-method approximation.
Method. At the first level, the mean is \((0.095+0.105)/2=0.100\). Each outcome differs from its mean by \(0.005\), so the variance is \((0.005^2+0.005^2)/2=0.000025\). At the second level, the mean is \((0.18+0.22)/2=0.200\), and the variance is \((0.02^2+0.02^2)/2=0.0004\).
These variances equal \(0.25\mu^4\) at both levels: \(0.25(0.100)^4=0.000025\), and \(0.25(0.200)^4=0.0004\). For \(h(y)=1/y\), \(h'(\mu)=-1/\mu^2\), so the approximate transformed variance is \(\operatorname{Var}(Y)/\mu^4=0.25\) at each level.
For the first level, the transformed outcomes are \(1/0.095=10.526316\) and \(1/0.105=9.523810\). With two equally likely values, the variance is one quarter of their squared difference: \((10.526316-9.523810)^2/4=0.2513\), to four decimal places.
For the second level, the transformed outcomes are \(1/0.18=5.555556\) and \(1/0.22=4.545455\). Their variance is \((5.555556-4.545455)^2/4=0.2551\), to four decimal places. Both exact variances are close to the delta-method value \(0.25\), but not identical.
Check. The response is positive at every possible outcome, as required by the reciprocal transformation. The transformed variance is nearly constant across the two levels. The common mistake is to omit the square on the derivative: variance is multiplied by \([h'(\mu)]^2\), not by \(h'(\mu)\).
Worked Example: Inverting a Square-Root Fitted Value
In a hypothetical regression, let \(Z=\sqrt{Y}\). At a fixed predictor value, suppose \(Z\) takes the values \(2.5\) and \(3.5\), each with probability \(1/2\). Find the fitted value obtained by inverting \(E[Z\mid X=x]\) and the actual conditional mean of \(Y\).
Method. The transformed-scale mean is \(E[Z\mid X=x]=(2.5+3.5)/2=3\). Direct inversion gives \(3^2=9\). But \(Y=Z^2\), so calculate the original-scale mean from the outcomes: \(2.5^2=6.25\) and \(3.5^2=12.25\). Their average is \((6.25+12.25)/2=9.25\).
The difference also follows from \(\operatorname{Var}(Z\mid X=x)=((2.5-3)^2+(3.5-3)^2)/2=0.25\) and the identity \(E[Z^2]=[E(Z)]^2+\operatorname{Var}(Z)\). Therefore \(E[Y\mid X=x]=3^2+0.25=9.25\).
Check. The inverse-transformed fitted value is \(9\), whereas the conditional mean is \(9.25\). They differ because squaring is nonlinear and the transformed response has nonzero conditional variance.
Interpreting Fitted Values After Transformation
A regression fitted to \(h(Y)\) produces fitted values on the transformed scale. Applying \(h^{-1}\) gives a value on the original scale, but in general
Equality may hold in special cases or under particular interpretations, but it should not be assumed. With a log response and conditionally normal log errors, exponentiating the fitted log mean gives the conditional median; the conditional arithmetic mean requires the factor \(\exp(\tau^2/2)\). With a square-root response, if \(Z=\sqrt{Y}\), then \(E[Y\mid X=x]=[E(Z\mid X=x)]^2+\operatorname{Var}(Z\mid X=x)\). For a reciprocal response, there is no general rule that turns \(E[1/Y\mid X=x]\) into \(E[Y\mid X=x]\) by taking a reciprocal.
Consequently, state clearly which quantity a prediction represents. If the objective is an original-scale conditional mean, use a justified distributional model or an appropriate retransformation adjustment. Do not apply a correction without its assumptions: the lognormal correction, for example, relies on normality and a specified log-scale variance.
Limitations and Exam Traps
- Choosing by appearance alone. A fan-shaped residual pattern does not identify a unique transformation. Consider whether the variance grows like \(\mu\), \(\mu^2\), or another pattern, and check the transformed residuals.
- Confusing response and predictor transformations. Transforming \(Y\) changes the quantity modeled as the response. Transforming a predictor changes the model's functional form in that predictor; these are different choices.
- Ignoring the response domain. A logarithm and reciprocal require positive responses. A square root requires nonnegative responses. Adding a constant to accommodate zero or negative values changes the transformation and should be justified.
- Expecting variance stabilization to remove curvature. A variance transformation targets spread. It does not ensure that the conditional mean is correctly modeled as linear in the chosen predictors.
- Reversing the variance relation. If variance is proportional to the mean, the square-root transformation is the standard candidate; a log transformation is motivated by variance proportional to the square of the mean.
- Calling an approximation exact. The delta method is a local, first-order approximation. Its accuracy depends on the response distribution and how concentrated the response is around its conditional mean.
In practice, compare residual spread before and after fitting the transformed-response model, while also checking that the mean pattern remains reasonable. The goal is not to make a transformation for its own sake; it is to obtain a defensible model and to report predictions on the scale that answers the question.
Check Your Understanding
For each problem, identify the transformation or prediction quantity requested and keep approximate results distinct from exact ones.
- A hypothetical count has a Poisson distribution with mean \(36\) at a specified predictor value. Using the delta method, what is the approximate variance of its square-root transform? (Answer: \(0.25\))
- A positive response has conditional variance \(\sigma^2=0.16\mu^2\). Which standard transformation is motivated by this pattern, and what is the approximate transformed variance? (Answer: \(\log Y\); approximately \(0.16\))
- A positive response has conditional variance \(0.09\mu^4\). Using \(h(Y)=1/Y\), what is the delta-method approximate conditional variance after transformation? (Answer: \(0.09\))
- At a predictor value, suppose \(\log Y\mid X=x\sim N(\log 50,0.04)\). Find the inverse-transformed fitted log mean and the original-scale conditional mean, rounded to the nearest cent. (Answer: fitted value \(=\$50.00\); conditional mean \(=50e^{0.02}=\$51.01\))
- At a predictor value, let \(Z=\sqrt{Y}\) take values \(1.5\) and \(2.5\), each with probability \(1/2\). Find the inverse of the transformed-scale mean and the original-scale conditional mean. (Answer: inverse-transformed value \(=4\); conditional mean \(=(2.25+6.25)/2=4.25\))