Why Transformations and Collinearity Must Be Considered Together
A regression can fit poorly because its mean relationship is curved, because predictors carry overlapping information, or because both issues occur at once. These problems require different responses. Adding a squared predictor can represent curvature; it does not automatically solve collinearity. Transforming the response changes the scale on which errors and fitted values are described; it does not guarantee that the predictors are well behaved.
This integrated case compares a straight-line model with a quadratic specification, diagnoses overlapping predictors using a variance inflation factor, and contrasts a polynomial model with a log-response model. The calculations are designed to make the scale of every coefficient and residual explicit.
A quadratic specification includes both \(z\) and \(z^2\): \(\widehat{Y}=\widehat{\beta}_0+\widehat{\beta}_1z+\widehat{\beta}_2z^2\). A log-response specification instead models \(\log(Y)\) as a linear function of predictors and requires positive responses. These models encode different mean relationships and make coefficients mean different things.
Worked Example: Correcting Curvature With a Quadratic Term
Worked Example: Correcting Curvature With a Quadratic Term
A hypothetical insurer records a response score \(Y\) at five values of a centered exposure index \(z\). The observations are \((-2,19),(-1,7),(0,22),(1,7),(2,19)\). Compare the straight-line and quadratic least-squares fits, and verify the quadratic residual sum of squares. Treat the observations as a regression data set; no probability or mortality model is being assumed.
Method. Fit each model by least squares. Because the predictor values are symmetric around zero, the linear and quadratic columns can be handled with centered sums. The common mistake this avoids is guessing coefficients from a plausible-looking curve instead of minimizing RSS.
Straight line. The response mean is \(\overline{Y}=(19+7+22+7+19)/5=74/5=14.8\), and \(\overline{z}=0\). The slope numerator is \((-2)(19)+(-1)(7)+(1)(7)+(2)(19)=0\), so the fitted straight line is \(\widehat{Y}=14.8\). Its RSS is \(4.2^2+(-7.8)^2+7.2^2+(-7.8)^2+4.2^2=208.8\).
Quadratic fit. The average of \(z^2\) is \((4+1+0+1+4)/5=2\). By symmetry, the centered column \(z^2-2\) is orthogonal to the column \(z\), so the fitted coefficient of \(z\) is zero. The coefficient of \(z^2\) is the slope in the regression of \(Y\) on \(z^2-2\):
The intercept is chosen so the fitted values average to \(\overline{Y}\). Therefore \(\widehat{\beta}_0=14.8-2(9/7)=428/35\), and the fitted model is \(\widehat{Y}=428/35+(9/7)z^2\). At \(z=-2,-1,0,1,2\), the fitted values are \(608/35,473/35,428/35,473/35,608/35\), or approximately \(17.3714,13.5143,12.2286,13.5143,17.3714\).
The residuals, observed minus fitted, are \(57/35,-228/35,342/35,-228/35,57/35\), or approximately \(1.6286,-6.5143,9.7714,-6.5143,1.6286\). Thus \(RSS=(2(57^2)+2(228^2)+342^2)/35^2=227430/1225\approx185.6571\). The quadratic RSS is smaller than the straight-line RSS, as it must be: the straight-line model is nested within the quadratic model by setting \(\beta_2=0\).
As a check, the reduction in RSS is \(208.8-185.6571\approx23.1429\), which equals \(18^2/14=324/14\). The values \(16,2,3\) are not least-squares coefficients here: the candidate equation \(16+2z+3z^2\) gives fitted values \(24,17,16,17,24\), not the least-squares solution. The correct calculation gives \(\widehat{\beta}_1=0\), \(\widehat{\beta}_2=9/7\), and \(\widehat{\beta}_0=428/35\).
Why the Quadratic Calculation Works
The least-squares coefficient for a predictor is its association with the response after accounting for the other columns in the design matrix. Here, symmetry makes the centered columns \(z\) and \(z^2-2\) orthogonal: \(\sum_i z_i(z_i^2-2)=0\). Consequently, estimating the coefficient of \(z^2\) does not require adjusting for a nonzero linear coefficient, and the ordinary one-predictor slope formula applies to \(z^2-2\).
The intercept follows from the least-squares normal equation for the constant column: residuals sum to zero. Since the average of \(z^2\) is 2, the average fitted value is \(\beta_0+2\beta_2\), which must equal \(\overline{Y}\). This gives \(\widehat{\beta}_0=\overline{Y}-2\widehat{\beta}_2\). This shortcut depends on the stated symmetry and centering; in a general design, use the full least-squares equations.
Worked Example: Diagnosing Collinearity in Two Measures
Worked Example: Diagnosing Collinearity in Two Measures
A hypothetical risk study has four observations. Let \(A\) be a centered exposure measure, and let \(B_c\) be a second measure centered by subtracting 20 from its original values. The data are shown below. Fit the stated two-predictor model and assess collinearity using the VIFs. For this calculation, use the usual independent, constant-variance error model; the very small sample makes the uncertainty estimates illustrative rather than a basis for broad conclusions.
| Observation | \(A\) | \(B_c\) | \(Y\) |
|---|---|---|---|
| 1 | -3 | -5.5 | 100.5 |
| 2 | -1 | -3.5 | 100.5 |
| 3 | 1 | 3.5 | 97.5 |
| 4 | 3 | 5.5 | 101.5 |
Method. First calculate the predictor correlation and VIF; then verify the fitted coefficients and residual variance. This avoids treating a high VIF as evidence that the overall fitted values must be poor.
Both predictor means are zero. The sums of squares and cross-products are \(\sum A_i^2=9+1+1+9=20\), \(\sum B_{c,i}^2=30.25+12.25+12.25+30.25=85\), and \(\sum A_iB_{c,i}=16.5+3.5+3.5+16.5=40\). Therefore \(r_{A,B_c}=40/\sqrt{20(85)}\), so \(r^2=1600/1700=16/17\). With two predictors, the VIF for either predictor is \(1/(1-r^2)=17\). This is substantial predictor overlap.
The fitted equation is \(\widehat{Y}=100+2A-B_c\). Its fitted values are \(99.5,101.5,98.5,100.5\), giving residuals \(1,-1,-1,1\). These residuals sum to zero and have zero cross-product with both predictors: \(\sum A_i e_i=-3+1-1+3=0\) and \(\sum B_{c,i}e_i=-5.5+3.5-3.5+5.5=0\). This verifies the least-squares normal equations. The RSS is \(1+1+1+1=4\); with \(n=4\) observations and \(p+1=3\) fitted coefficients, the residual variance estimate is \(s^2=4/(4-3)=4\).
For a coefficient of a predictor \(X_j\), its variance is \(s^2/\{\sum_i(X_{ij}-\overline{X}_j)^2(1-R_j^2)\}\), where \(R_j^2\) is from regressing that predictor on the other predictor. Here \(R_j^2=16/17\). Thus \(\operatorname{Var}(\widehat{\beta}_A)=4/[20(1/17)]=3.4\), giving standard error \(\sqrt{3.4}\approx1.844\). For \(B_c\), the variance is \(4/[85(1/17)]=0.8\), giving standard error \(\sqrt{0.8}\approx0.894\). The estimates are identifiable, but their conditional uncertainty is inflated by the overlap between predictors.
The VIF does not say that the fitted values are necessarily inaccurate, and it does not diagnose curvature. It indicates that it is difficult to separate the predictors’ individual contributions. Centering predictors can change the interpretation of the intercept and can reduce nonessential correlation in some models; it does not eliminate the shared information represented by a high correlation between these two measures.
Worked Example: Comparing a Log Response With a Polynomial
Worked Example: Comparing a Log Response With a Polynomial
A hypothetical benefit study records positive responses \(Y=100,110,121\) at predictor values \(x=0,1,2\). Compare a straight-line fit on the original response scale, a quadratic fit on that scale, and a straight-line fit to \(\log(Y)\). Then interpret a prediction at \(x=3\). For the final back-transformation calculation, assume a normal error on the log scale with known variance \(\sigma^2=0.04\).
Method. Fit each specification on the scale named in its equation, and only then translate predictions back if needed. This avoids interpreting a log-scale slope as a dollar increase or treating \(\exp(\widehat{\eta})\) as the conditional arithmetic mean without an error-distribution assumption.
For the original-scale straight line, \(\overline{x}=1\) and \(\overline{Y}=331/3\). The slope is \(\sum_i(x_i-\overline{x})(Y_i-\overline{Y})/\sum_i(x_i-\overline{x})^2=21/2=10.5\), and the intercept is \(331/3-10.5=599/6\). Its fitted values are \(599/6,331/3,725/6\); the residuals are \(1/6,-1/3,1/6\), and \(RSS=1/6\).
For the quadratic fit, the first differences in \(Y\) are 10 and 11, so the second difference is 1. For equally spaced \(x\), the quadratic coefficient is half the second difference, or \(0.5\). Substitution at \(x=0\) gives intercept 100, and the value at \(x=1\) gives linear coefficient 9.5. Thus \(\widehat{Y}=100+9.5x+0.5x^2\), which fits all three observations exactly and has RSS 0. This exact fit uses three coefficients for three observations, leaving no residual degrees of freedom to estimate error variance.
On the log scale, the observations satisfy \(Y=100(1.1)^x\), so \(\log(Y)=\log(100)+x\log(1.1)\) exactly. The fitted log-scale intercept is \(\log(100)\), and the slope is \(\log(1.1)\approx0.09531\). A one-unit increase in \(x\) multiplies the fitted geometric level by \(\exp(0.09531)=1.1\), a 10% increase. It is not an increase of 0.09531 dollars or 9.531 dollars.
At \(x=3\), the log-linear model gives a fitted log value \(\log(100)+3\log(1.1)\), corresponding to \(100(1.1)^3=\$133.10\) when exponentiated. The quadratic model gives \(100+9.5(3)+0.5(9)=\$133.00\). Both fit the three observed points perfectly, yet they imply different functional forms and are not thereby validated for extrapolation.
Under the explicitly assumed normal log-scale error, exponentiating the fitted log value gives the conditional median, not the conditional arithmetic mean. The latter is \(\exp(\eta+\sigma^2/2)\), where \(\eta\) is the fitted log-scale mean. At \(x=3\), it is \(133.10\exp(0.04/2)=133.10\exp(0.02)\approx\$135.79\). This correction depends on the lognormal assumption and the stated variance; it is not a universal adjustment for every transformed regression.
Comparing Specifications and Reading Diagnostics
A transformation should be chosen to represent the relationship and error behavior that are appropriate for the problem, not merely because it produces a smaller numerical RSS. RSS values on different response scales are not directly comparable: a log-scale RSS is a sum of squared log residuals, while a raw-scale RSS is a sum of squared response residuals. Their units and loss functions differ.
| Specification | What it represents | Coefficient interpretation | Important limitation |
|---|---|---|---|
| Quadratic response model | Additive curvature in the conditional mean on the original response scale | \(\beta_2\) is the coefficient of \(x^2\), in response units per squared predictor unit | \(x\) and \(x^2\) can be highly correlated; extrapolation may be unstable or implausible |
| Log-response linear model | A linear relationship in log response; exponentiation gives a multiplicative fitted level | A one-unit predictor increase changes the fitted geometric level by a factor \(\exp(\beta_j)\) | Requires positive responses; conditional-mean back-transformation requires an error-distribution adjustment |
| Multiple regression with overlapping predictors | Conditional effects of each predictor while holding the others fixed | Each slope describes a conditional association, not necessarily a marginal association | High VIF can make individual slopes imprecise even when the overall fit is useful |
Residual patterns help distinguish problems that a single summary cannot. Curvature in residuals plotted against a predictor suggests that the mean form may be inadequate. A fan-shaped spread can indicate nonconstant variance on the scale being modeled. A high VIF or unstable coefficient estimates indicate predictor overlap, not necessarily a residual pattern. Check both the response-scale fit and the predictor design before concluding that a transformation has solved the problem.
This integrated way of comparing functional form, coefficient meaning, and predictor dependence is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus. Keep the conclusions limited to what each calculation supports: a lower nested-model RSS demonstrates improved in-sample fit, a VIF diagnoses coefficient uncertainty from overlap, and neither alone establishes that a specification will predict well on new data.
Common Traps
- Guessing coefficients instead of fitting. A plausible curve is not a least-squares calculation. Verify estimates by checking the normal equations or recomputing residuals and RSS.
- Confusing curvature with collinearity. A quadratic term may model curvature but can itself be correlated with the linear term. A high VIF concerns uncertainty in conditional slopes.
- Comparing RSS across different response scales. Raw-response and log-response residual sums of squares measure different discrepancies and cannot be ranked directly as if they had the same units.
- Back-transforming without stating the target. Under normal log-scale errors, exponentiating the fitted log mean gives a conditional median; the conditional arithmetic mean includes the factor \(\exp(\sigma^2/2)\).
- Trusting a saturated fit. A three-parameter quadratic fitted to three observations has zero RSS but no residual degrees of freedom. That exact fit does not provide an estimate of residual variability or evidence of reliable extrapolation.
Check Your Understanding
For each problem, identify the relevant scale or diagnostic before calculating.
- A log-response model has slope \(0.08\) for a one-unit increase in a rating index. What multiplicative change in the fitted geometric response does this represent? Give the percentage to two decimals. (Answer: \(\exp(0.08)-1\approx 8.33\%\).)
- Two centered predictors have correlation \(0.95\). Calculate their VIF in a two-predictor regression. Give the result to four decimals. (Answer: \(1/(1-0.95^2)=10.2564\).)
- At a particular covariate value, a log-scale model gives fitted log mean \(\eta=4.2\), with normally distributed log errors of variance \(0.09\). State the conditional median and conditional arithmetic mean on the original scale. (Answer: Median \(=\exp(4.2)\); mean \(=\exp(4.2+0.09/2)=\exp(4.245)\).)
- A least-squares quadratic model has RSS \(72\), and the straight-line model using the same response and observations has RSS \(90\). What is the quadratic model’s reduction in RSS, and is the direction consistent with nesting? (Answer: Reduction \(=90-72=18\); yes, the quadratic RSS is lower, as required for nested least-squares models.)
- A raw-scale quadratic fit gives \(\widehat{Y}=40+3x+0.4x^2\). Calculate its fitted value at \(x=5\), and explain whether that value is automatically the conditional mean under a lognormal model. (Answer: \(\widehat{Y}=40+15+10=65\); no, this is a raw-scale quadratic prediction, not a lognormal back-transformation.)