Square-Root and Reciprocal Transformations for Curved Patterns

Why Transform a Curved Relationship?

A straight-line regression can be a poor description of a relationship that bends systematically. For example, a response may rise increasingly quickly as a predictor increases, or it may decrease rapidly at first and then level off. A high \(R^2\) alone does not establish that a straight-line model is appropriate: a plot of residuals can still reveal a curved pattern.

A transformation changes the scale on which a variable is represented. If it makes the relationship approximately linear, we can fit a linear regression to the transformed data. The transformation is part of the model: it determines what is predicted, what units the residuals have, and what pattern should be checked.

Definition: A square-root transformation replaces a nonnegative variable \(v\) by \(\sqrt{v}\). A reciprocal transformation replaces a nonzero variable \(v\) by \(1/v\). In regression, either transformation can be applied to the response, the predictor, or both. A fitted line after transformation is linear in the variables actually used in the regression, even though its relationship on the original scales may be curved.

These transformations require care with the values they can accept. A square root is defined for nonnegative values; a reciprocal is undefined at zero. This tutorial focuses on positive variables, so both transformations are defined. If a reciprocal response is modeled, the fitted reciprocal values must also be nonzero to convert them back.

Models and Residuals on the Chosen Scale

Before fitting a transformed regression, specify its model and assumptions. Let \(x_i\) and \(y_i\) denote the predictor and response for observation \(i\). For a transformed response \(g(y)\) and transformed predictor \(h(x)\), a linear regression model has the form

g(y_i)=\beta_0+\beta_1 h(x_i)+\varepsilon_i.

The usual regression assumptions apply to the variables on the modeled scale: the conditional mean of \(g(y)\) is linear in \(h(x)\); observations are independent; the errors have mean zero and constant variance; and, when normal-based inference is needed, the errors are approximately normal. A transformation does not guarantee that these conditions hold. The data collection design is the main basis for assessing independence, while residual patterns help assess whether the modeled relationship and error spread are plausible.

Ordinary least squares chooses the intercept and slope to minimize the sum of squared residuals on the response scale used in the model. A residual is the observed response on that scale minus its fitted value. Thus, if the model response is \(\sqrt{y}\), its residuals are in square-root units, whereas residuals from an untransformed model are in the original units of \(y\).

e_i=g(y_i)-\widehat{g(y_i)},\qquad \widehat{g(y_i)}=b_0+b_1h(x_i).

Here \(b_0\) and \(b_1\) are the least-squares estimates of the intercept and slope. In a model that transforms only the predictor, the response remains on its original scale, so the residuals retain the original response units.

Three Useful Model Forms

The following equations show how common transformations alter the relationship on the original scale. They are algebraic descriptions of candidate forms, not guarantees that a particular data set follows them.

Regression fitted	Equivalent form on the original scales	Typical pattern represented
\(\sqrt{y}=\beta_0+\beta_1x+\varepsilon\)	The fitted response has form \((\beta_0+\beta_1x)^2\), when the fitted square root is nonnegative.	A response that curves upward or downward with a quadratic-like shape.
\(1/y=\beta_0+\beta_1x+\varepsilon\)	The fitted response has form \(1/(\beta_0+\beta_1x)\), where the denominator is nonzero.	A response whose reciprocal changes approximately linearly with \(x\).
\(y=\beta_0+\beta_1(1/x)+\varepsilon\)	The fitted response has form \(\beta_0+\beta_1/x\), for \(x>0\).	A response that changes sharply at smaller \(x\), then tends to level off.

When the response is transformed, the fitted coefficients describe change on that transformed response scale. For example, in a model of \(\sqrt{y}\) against \(x\), a one-unit increase in \(x\) corresponds to a change of \(\beta_1\) in the modeled square-root response, not a constant change of \(\beta_1\) in \(y\). Similarly, a reciprocal-response slope describes change in \(1/y\), not in \(y\). Do not interpret these slopes as ordinary-unit changes in the response.

Worked Examples

Worked Example: A Square-Root Response

Suppose a hypothetical process produces the following positive response values at five predictor settings. Consider whether modeling the square root of the response produces a better-behaved fit than a line in the original response.

\(x\)	\(y\)	\(\sqrt{y}\)
1	9	3
2	16	4
3	25	5
4	36	6
5	49	7

All responses are positive, so the square-root transformation is defined. Treat the observations as independent for this illustration. The transformed values follow \(\sqrt{y}=2+x\) exactly, so a least-squares fit on the transformed scale is

\widehat{\sqrt{y}}=2+1x.

The fitted transformed values are 3, 4, 5, 6, and 7; the transformed residuals are all zero. The corresponding fitted responses after squaring are 9, 16, 25, 36, and 49. These match the observed responses, although in general a squared fitted square root need not give an unbiased estimate of the conditional mean of \(y\).

For comparison, fit an ordinary line to \(y\) against \(x\). The means are \(\bar{x}=3\) and \(\bar{y}=27\), and the least-squares slope is

b_1=\frac{\sum (x_i-\bar{x})(y_i-\bar{y})}{\sum (x_i-\bar{x})^2} =\frac{(-2)(-18)+(-1)(-11)+(0)(-2)+(1)(9)+(2)(22)} {(-2)^2+(-1)^2+0^2+1^2+2^2} =\frac{100}{10}=10.

The intercept is \(b_0=\bar{y}-b_1\bar{x}=27-10(3)=-3\), so the original-scale fitted line is \(\hat{y}=-3+10x\). Its fitted values are 7, 17, 27, 37, and 47. The residuals \(y-\hat{y}\) are 2, \(-1\), \(-2\), \(-1\), and 2. Their systematic positive, negative, then positive pattern is consistent with curvature rather than random scatter around a line. The transformed fit removes that pattern for these deliberately exact data.

Worked Example: A Reciprocal Response

Consider hypothetical positive responses measured at \(x=0,1,2,3\): \(y=1,\frac12,\frac13,\frac14\). Compare a line in the original response with a line in the reciprocal response. The response values are all nonzero, and the reciprocal values are 1, 2, 3, and 4.

The transformed values satisfy \(1/y=1+x\). The fitted reciprocal-response model is therefore

\widehat{1/y}=1+1x, \qquad \hat{y}=\frac{1}{1+x}.

The reciprocal-scale residuals are all zero. The inverse expression gives the fitted response values \(1,\frac12,\frac13,\frac14\), respectively. It also makes the domain restriction clear: \(1+x\) must not be zero when converting a fitted reciprocal back to a response.

For the untransformed regression, \(\bar{x}=1.5\) and \(\bar{y}=25/48\). The slope calculation is

b_1= \frac{(-1.5)(1)+(-0.5)(1/2)+(0.5)(1/3)+(1.5)(1/4)} {(-1.5)^2+(-0.5)^2+(0.5)^2+(1.5)^2} =\frac{-29/24}{5}=-\frac{29}{120}\approx -0.2417.

The intercept is \(b_0=25/48-(-29/120)(1.5)=53/60\approx0.8833\). This gives fitted values approximately 0.8833, 0.6417, 0.4000, and 0.1583. Subtracting these from the observed values gives original-scale residuals approximately 0.1167, \(-0.1417\), \(-0.0667\), and 0.0917. The positive residuals at both ends and negative residuals in the middle suggest a curved pattern. The reciprocal model captures the pattern in these data. Its zero residuals are on the reciprocal scale, so their magnitudes should not be compared directly with the original-scale residuals.

Worked Example: A Reciprocal Predictor

Suppose a hypothetical response is observed at positive predictor values \(x=1,2,3,4,6\), with responses \(y=8,5,4,3.5,3\). Consider a reciprocal transformation of the predictor while leaving the response unchanged. The transformed predictor values are \(1,\frac12,\frac13,\frac14,\frac16\), and in these data the model is exactly \(y=2+6(1/x)\).

The fitted model on the transformed predictor scale is

\hat{y}=2+6(1/x).

It gives fitted responses 8, 5, 4, 3.5, and 3, so the residuals are all zero, in the original response units. The predictor values are strictly positive, as required for \(1/x\). The relation is linear in \(1/x\), but not in \(x\).

For comparison, the ordinary line in \(x\) has \(\bar{x}=3.2\), \(\bar{y}=4.7\), and

b_1=\frac{\sum (x_i-\bar{x})(y_i-\bar{y})}{\sum (x_i-\bar{x})^2} =\frac{-12.54}{14.8}\approx-0.8473, \qquad b_0=4.7-(-0.8473)(3.2)\approx7.411.

The fitted line is \(\hat{y}=7.411-0.8473x\). At \(x=1,2,3,4,6\), its fitted values are approximately 6.564, 5.717, 4.869, 4.022, and 2.328. The residuals are approximately 1.436, \(-0.717\), \(-0.869\), \(-0.522\), and 0.672. Their pattern—positive at the ends and negative in between—indicates that the straight-line fit misses curvature. Because this model leaves \(y\) untransformed, its residuals have the same units as \(y\).

Why the Transformations Produce Curves

The curvature represented by these models follows directly from reversing the transformation. If \(\sqrt{y}=\beta_0+\beta_1x\) describes the fitted relationship and the fitted square root is nonnegative, squaring gives

y=(\beta_0+\beta_1x)^2 =\beta_0^2+2\beta_0\beta_1x+\beta_1^2x^2.

The \(x^2\) term explains why a straight line in \(\sqrt{y}\) can describe a curved pattern in \(y\). Likewise, from \(1/y=\beta_0+\beta_1x\), taking reciprocals gives

y=\frac{1}{\beta_0+\beta_1x}.

This is generally curved and is only defined where the denominator is nonzero. A reciprocal predictor instead gives \(y=\beta_0+\beta_1/x\), also a curved relationship for positive \(x\). These algebraic forms explain what the models can represent; residuals are still needed to assess whether they describe the observed data adequately.

Comparing Fits Responsibly

A useful comparison asks whether the transformed fit removes a systematic residual pattern, not merely whether a numerical summary becomes larger or smaller. For each candidate model, examine residuals against the predictor or fitted values on the scale used for that model. Curvature, a widening or narrowing spread, or isolated unusual residuals can remain after transformation. A transformation may straighten the mean relationship while leaving nonconstant error variance, or it may fail to make the relationship approximately linear.

Do not compare residual magnitudes, residual standard errors, or \(R^2\) values from models with different response scales as though they were directly interchangeable. For instance, an error of 0.2 in \(\sqrt{y}\) is not the same size or unit as an error of 0.2 in \(y\). If prediction quality on the original response scale is the goal, evaluate predictions on that scale using a suitable comparison, while remembering that simply reversing a transformation does not generally recover the conditional mean exactly.

A transformation also changes the meaning of the fitted coefficients. In a square-root-response model, the slope is a change in fitted square-root response per unit increase in \(x\). In a reciprocal-response model, it is a change in fitted reciprocal response. In a reciprocal-predictor model, it is a change in \(y\) per unit increase in \(1/x\). State the scale and units when interpreting any coefficient.

Common Mistake

Common pitfall: “The transformed model has smaller residuals, so it must be better.” This reasoning compares residuals measured on different response scales. A residual on the square-root or reciprocal scale cannot be judged against one on the original scale just by its numerical size. Instead, inspect whether residuals have a systematic pattern on each model's own scale, check the assumptions for that scale, and consider the original-scale prediction goal separately.

Another error is applying \(1/x\) when some predictor values are zero, or \(1/y\) when a response is zero. The reciprocal is undefined at zero, so such a model cannot be calculated as written for those observations. Do not silently discard or alter values to make the transformation possible; the modeling choice must be justified for the data and question.

Key takeaway: Square-root and reciprocal transformations can represent curved relationships by making a transformed response or predictor approximately linear. Fit the model on its stated scale, calculate residuals on that same response scale, and judge the residual pattern without confusing units or coefficient meanings.

Check Your Understanding

Use the definitions and examples to answer these questions.

In a model of \(\sqrt{y}\) against \(x\), what units do the residuals have, and what change does the slope describe?
Why can a straight line in \(1/y\) against \(x\) represent a curved relationship between \(y\) and \(x\)?
A predictor includes the value zero. Can it be used unchanged in a reciprocal-predictor model? Explain.
Why is it misleading to declare a transformed model better solely because its residual standard error is numerically smaller than that of an untransformed model?
For a model \(y=\beta_0+\beta_1(1/x)+\varepsilon\), identify the scale of the residuals and the condition on \(x\) needed to calculate the predictor transformation.