Comparing Models Fit on Different Response Scales

Why the Response Scale Matters

Regression candidates are sometimes fitted to different versions of the same outcome. One model might predict $Y$, while another predicts $\log Y$ or $\sqrt{Y}$. Such transformations can be useful when they make a relationship easier to model or improve the behavior of residuals. But they change what the model’s residuals measure. A residual of 1 dollar is not the same quantity as a residual of 1 log-dollar, and the corresponding sums of squares are not directly comparable.

A fair comparison begins by identifying the target. If the practical goal is to predict the original outcome $Y$, compare predictions and errors on the $Y$-scale. If comparing likelihood-based criteria, put the likelihoods on the same response scale, including the required change-of-variables term. A smaller reported number is not automatically evidence of a better model when the numbers describe different scales.

Definition: For observations $y_1,\ldots,y_n$, a response-scale model is fitted to those outcomes, while a transformed-response model is fitted to $z_i=g(y_i)$. The residual sum of squares for a fitted response $w$ is $\sum_{i=1}^n(w_i-\hat w_i)^2$. It measures squared error in the units of $w$, so residual sums of squares from different response scales are not directly comparable.

Residual Sums of Squares and R-Squared

For a regression with an intercept, the usual coefficient of determination on response $w$ is $R^2=1-\mathrm{RSS}_w/\mathrm{SST}_w$, where $\mathrm{SST}_w=\sum_i(w_i-\bar w)^2$. Both the residual sum of squares and the total sum of squares are calculated using the response $w$. Consequently, $R^2$ describes the fraction of variation explained on that particular scale. It is not a scale-free score for comparing models fitted to different responses.

A small residual sum of squares on a log scale may be numerically much smaller than one on the original scale simply because logarithms have different units and compress large values. Likewise, a high $R^2$ for $\log Y$ does not establish that the log-response model predicts $Y$ more accurately than a model fitted directly to $Y$. To answer that predictive question, obtain predictions on the original scale and compare them with the observed outcomes there.

Worked Example: Comparing Fit on the Original and Log Scales

A hypothetical analyst has four positive outcomes, $y=(2,4,8,16)$, and fits two regression candidates with intercepts. Candidate A is fitted to $Y$ and predicts $(3,3,9,15)$. Candidate B is fitted to $\log Y$ and gives back-transformed point predictions $(2.5,3.5,8.5,15)$. For this comparison, use the listed original-scale predictions as the point forecasts. Determine which has lower original-scale squared error, and explain why the fitted-scale residual statistics should not be compared directly.

Method and scale check. Candidate A’s residuals are $2-3=-1$, $4-3=1$, $8-9=-1$, and $16-15=1$. Thus its original-scale residual sum of squares is $1+1+1+1=4$.

Candidate B’s log-scale fitted values are $(\log 2.5,\log 3.5,\log 8.5,\log 15)$. Its log-scale residuals, rounded to four decimals, are $(-0.2231,0.1335,-0.0606,0.0645)$. Squaring and summing gives approximately $0.0498+0.0178+0.0037+0.0042=0.0755$. This $0.0755$ is squared log-scale error, not squared outcome-scale error; it cannot be ranked against Candidate A’s $4$.

For a fair squared-error comparison on the original scale, Candidate B’s prediction errors are $2-2.5=-0.5$, $4-3.5=0.5$, $8-8.5=-0.5$, and $16-15=1$. Its squared error is $0.25+0.25+0.25+1=1.75$. Therefore Candidate B has the lower original-scale sum of squared prediction errors: $1.75<4$. This conclusion follows from the common-scale errors, not from comparing $0.0755$ with $4$.

For reference, the original-scale mean is $7.5$, and its total sum of squares is $30.25+12.25+0.25+72.25=115$. Candidate A therefore has original-scale $R^2=1-4/115\approx0.9652$. On the log scale, $\overline{\log y}\approx1.7329$, $\mathrm{SST}_{\log y}\approx2.4023$, so Candidate B’s log-scale $R^2$ is approximately $1-0.0755/2.4023=0.9686$. These values describe different response scales; their numerical proximity does not make them directly comparable.

Common mistake avoided: treating Candidate B’s smaller numerical residual sum of squares as evidence by itself. The units differ, so compare errors only after both sets of predictions and observations are on the same scale.

Information Criteria and the Change of Variables

AIC is based on a model’s maximized likelihood, so comparing AIC values requires likelihoods for the same observed data on a common scale. A Gaussian regression fitted to $Z=g(Y)$ supplies a density for $Z$, not directly for $Y$. If $g$ is differentiable and one-to-one, the density for $Y$ is obtained by the change-of-variables formula:

f_Y(y)=f_Z(g(y))\left|g'(y)\right|.

For independent observations, the log-likelihood on the $Y$-scale is the fitted log-likelihood on the $Z$-scale plus $\sum_i\log|g'(y_i)|$. The correction depends on the observed outcomes, not on the regression residuals. It must be included before comparing AIC values across response transformations. For $g(y)=\log y$, with $y>0$, $g'(y)=1/y$, so the correction to the log-likelihood is $-\sum_i\log y_i$. Since $\mathrm{AIC}=-2\ell+2k$, the corresponding correction to AIC is $+2\sum_i\log y_i$.

Identify the fitted response.
Record whether the candidate likelihood is for $Y$, $\log Y$, or another transformation. Confirm that the same observations are being modeled.

Convert the likelihood if needed.
For a one-to-one differentiable transformation, add the log-Jacobian term $\sum_i\log|g'(y_i)|$ to the transformed-response log-likelihood.

Compare criteria on the common data scale.
Calculate AIC from the likelihood for $Y$, counting the fitted parameters consistently. Lower AIC is then preferred among the candidate likelihood models, subject to the assumptions behind them.

Worked Example: Correcting AIC for a Log Response

A hypothetical analyst models four positive claim amounts $y=(2,4,8,16)$. The reported Gaussian AIC for a regression of $Z=\log Y$, calculated as a likelihood for $Z$, is 20.00. The AIC for a Gaussian regression directly on $Y$ is 28.00. Both AIC values include their fitted parameter counts. Decide which model has lower AIC after expressing both likelihoods for the observed claim amounts $Y$.

Method. For $g(y)=\log y$, $g'(y)=1/y$. Thus the log-likelihood for $Y$ under the log-response model is $\ell_Y=\ell_Z-\sum_i\log y_i$, and its AIC is $\mathrm{AIC}_Y=\mathrm{AIC}_Z+2\sum_i\log y_i$.

The product of the four observations is $2\cdot4\cdot8\cdot16=1024$, so $\sum_i\log y_i=\log(1024)=6.9315$ to four decimals. The correction is $2(6.9315)=13.8630$. Therefore the log-response model’s AIC for $Y$ is $20.00+13.8630=33.8630$, or $33.86$ to two decimals. The direct-$Y$ model’s AIC remains $28.00$. On the common $Y$-scale likelihood comparison, the direct-$Y$ model has the lower AIC.

Common mistake avoided: selecting the log-response model by comparing $20.00$ directly with $28.00$. The former omits the Jacobian needed to express its likelihood as a density for the observed claim amounts.

For a Gaussian regression with maximum-likelihood variance estimate, the response-scale AIC can be written as $n\{\log(2\pi)+1+\log(\mathrm{RSS}/n)\}+2k$, where $k$ counts all estimated parameters, including the error variance under this convention. This expression uses the RSS for the response whose likelihood is being calculated. Applying it to $\log Y$ gives an AIC for $\log Y$; to compare it with a model for $Y$, the transformation correction is still necessary. AIC values from models using different observations, or likelihoods with incompatible specifications, also do not become comparable merely by applying this correction.

Back-Transforming Predictions

A fitted value on a transformed scale is not always the conditional mean on the original scale after applying the inverse transformation. For squared-error prediction of $Y$, the target is $E[Y\mid x]$, because the conditional mean minimizes expected squared error. If the log-response model assumes $\log Y\mid x\sim N(\mu(x),\sigma^2)$, then $\exp(\mu(x))$ is the conditional median, not the conditional mean.

Result: If $\log Y\mid x\sim N(\mu(x),\sigma^2)$, then $E[Y\mid x]=\exp\{\mu(x)+\sigma^2/2\}$, while the conditional median is $\exp\{\mu(x)\}$. The mean formula requires the stated conditional normal distribution and variance; it does not follow from knowing only the fitted value of $\log Y$.

The result follows from the moment-generating function of a normal variable. If $Z=\log Y\mid x\sim N(\mu(x),\sigma^2)$, then $Y=e^Z$. The normal moment-generating function is $E[e^{tZ}\mid x]=\exp\{t\mu(x)+t^2\sigma^2/2\}$. Setting $t=1$ gives $E[Y\mid x]=\exp\{\mu(x)+\sigma^2/2\}$. Setting $Z=\mu(x)$, the median of the symmetric normal distribution, gives the original-scale median $e^{\mu(x)}$.

Worked Example: Choosing a Log-Scale Prediction for Squared Error

Suppose a hypothetical regression for positive repair costs satisfies $\log Y\mid x\sim N(\mu(x),\sigma^2)$, with predicted $\mu(x)=\log(50)$ and $\sigma^2=0.25$. Find the conditional median and conditional mean repair cost. State which prediction is appropriate for squared-error loss.

Method. Under the stated lognormal model, the conditional median is $\exp(\mu(x))=\exp(\log 50)=50$. The conditional mean is $\exp\{\mu(x)+\sigma^2/2\}=\exp\{\log 50+0.25/2\}$.

Since $0.25/2=0.125$, the conditional mean is $50e^{0.125}$. Using $e^{0.125}\approx1.133148$, this is $56.6574$, or $\$56.66$ to the nearest cent. The conditional median is $\$50.00$. For squared-error prediction of the cost, use the conditional mean, $\$56.66$; for absolute-error prediction under this distribution, the conditional median is the relevant target.

Common mistake avoided: exponentiating the fitted log mean and calling the result the expected cost. Here $50$ is the median, and omitting the variance adjustment understates the mean.

When the transformed errors are not conditionally normal, the lognormal adjustment need not be valid. A possible alternative for log-response models is a smearing estimate: multiply $\exp(\hat\mu(x))$ by an estimate of $E[\exp(\varepsilon)\mid x]$, where $\varepsilon=\log Y-\mu(x)$. This expectation can be estimated from suitable training residuals; if the residual distribution changes with predictors, a single common adjustment may not suffice. For transformations such as the square root, simply squaring the fitted transformed response also does not generally recover the conditional mean, because $E[Z^2\mid x]$ is not in general $\{E[Z\mid x]\}^2$.

Common-Scale Prediction Error

For a predictive comparison, select a loss function that matches the objective, then calculate it using observed outcomes and predictions on the same response scale. For squared-error loss on $Y$, a standard measure is $\mathrm{MSE}=n^{-1}\sum_i(y_i-\hat y_i)^2$, with $\mathrm{RMSE}=\sqrt{\mathrm{MSE}}$. For absolute-error loss, use $\mathrm{MAE}=n^{-1}\sum_i|y_i-\hat y_i|$. These measures are meaningful across response transformations only after each candidate’s prediction has been appropriately returned to the original scale.

When estimating predictive performance by cross-validation, use the same folds for all candidates. Each fold’s transformation, model fitting, and any bias correction must be estimated using that fold’s training data only; otherwise information from the held-out outcomes leaks into the prediction process. Evaluate all held-out predictions against the actual $Y$ values using the same loss function. This comparison addresses original-scale predictive accuracy, while corrected AIC addresses relative likelihood fit with a complexity penalty. They answer related but distinct questions.

Worked Example: Comparing Cross-Validated Errors on the Original Scale

A hypothetical analyst evaluates two regression candidates on the same three held-out cases. The observed costs are $(10,20,40)$. Candidate A is fitted to $Y$ and predicts $(12,18,38)$. Candidate B uses a transformed response; its predictions have already been inverse-transformed and bias-adjusted for the original-scale mean. Candidate B predicts $(11,22,36)$. Compare their original-scale RMSEs and MAEs.

Method. Evaluate each forecast against the observed cost, not against the transformed cost. Candidate A’s errors are $10-12=-2$, $20-18=2$, and $40-38=2$. Its squared errors sum to $4+4+4=12$, so its MSE is $12/3=4$ and its RMSE is $\sqrt{4}=2.00$. Its absolute errors sum to $2+2+2=6$, giving MAE $6/3=2.00$.

Candidate B’s errors are $10-11=-1$, $20-22=-2$, and $40-36=4$. Its squared errors sum to $1+4+16=21$, so its MSE is $21/3=7$ and its RMSE is $\sqrt7\approx2.65$ to two decimals. Its absolute errors sum to $1+2+4=7$, giving MAE $7/3\approx2.33$ to two decimals. Candidate A has the lower RMSE and MAE on these held-out original-scale outcomes.

Common mistake avoided: comparing Candidate B’s validation error on the transformed response with Candidate A’s error in cost units. The common-scale calculations show the predictive comparison for the stated original-cost target.

Exam Traps and Practical Checks

Ranking raw RSS values across transformations. First check the response used to calculate each RSS. Different response scales mean different error units.
Treating $R^2$ as scale-free. Each $R^2$ uses the total variation of its own response. Compare predictive performance on a chosen common scale instead.
Comparing unadjusted AIC values. A transformed-response likelihood is a density for the transformed observation. Include the change-of-variables term to express it as a density for the original observation.
Using the inverse transform without checking the target. Under squared-error loss, the target is the conditional mean. For a lognormal response, exponentiating the fitted log mean gives the median; the mean includes the variance adjustment.
Letting validation information affect fitting. Estimate transformations, fitted coefficients, and any retransformation adjustment within each training fold, then score the untouched held-out responses.

Key takeaway: Before ranking regression candidates, name the response scale and prediction target. Residual sums of squares and $R^2$ belong to their fitted scales; likelihood criteria need a common-scale likelihood; predictive errors should be computed from comparable predictions and outcomes on the target scale.

Check Your Understanding

For each question, identify the relevant scale before carrying out the comparison.

A model fitted to $Y$ has RSS $=18$, while a model fitted to $\log Y$ has RSS $=0.9$. Can the smaller reported RSS identify the better original-scale predictor? (Answer: No; the RSS values are on different response scales.)
For positive observations $y=(3,5,10)$, a log-response model has AIC $=16.0$ calculated for $Z=\log Y$. What is its AIC for $Y$, using the same parameter count? (Answer: $16+2\log(150)\approx26.02$.)
Under $\log Y\mid x\sim N(\log 80,0.09)$, what are the conditional median and mean of $Y$, to the nearest cent? (Answer: Median $\$80.00$; mean $80e^{0.045}\approx\$83.68$.)
On common held-out cases, the observed values are $(5,10,15)$. Model A predicts $(6,9,15)$, and Model B predicts $(5,12,14)$. Which has lower original-scale RMSE, and what is it to two decimals? (Answer: Model A; its RMSE is $\sqrt{(1+1+0)/3}\approx0.82$, versus Model B’s $\sqrt{(0+4+1)/3}=1.29$.)
A candidate’s transformed-scale $R^2$ is $0.94$, compared with $0.91$ for a model fitted directly to $Y$. What additional comparison is needed before concluding which better predicts $Y$ under squared-error loss? (Answer: Obtain appropriately calibrated predictions for both on the $Y$-scale and compare their squared errors on the same held-out observations.)

What You'll Learn