Tutorials › Exam SRM - Statistics for Risk Modeling › Comparing Likelihood Criteria After Response Transformations

Home  /  Tutorials  /  Exam SRM / Information Criteria

Exam SRM · Information Criteria · Tutorial 201 of 926

Comparing Likelihood Criteria After Response Transformations

Use the transformation’s Jacobian to put likelihoods for differently transformed responses on the same scale before comparing information criteria.

Intermediate 10 min read

What You'll Learn

  • Explain why likelihoods on different response scales are not directly comparable
  • Derive the Jacobian adjustment from the change-of-variables formula
  • Calculate adjusted log likelihoods, AIC, and BIC for transformed responses
  • Distinguish comparisons across transformations from comparisons using the same transformation
  • Recognize when a Jacobian is parameter-dependent or a simple adjustment does not apply

Why Response Transformations Affect Likelihood Comparisons

A response transformation can make a regression relationship easier to model. For example, a model may assume that the logarithm of a positive response is normally distributed, while another model assumes that the response itself is normally distributed. Both models may use the same observed cases, but their likelihoods are initially expressed as densities on different scales. Comparing those likelihoods without an adjustment can therefore give a misleading result.

The adjustment is the Jacobian from the change-of-variables formula. It converts a density for the transformed response into a density for the original response. Once the log likelihoods are expressed for the same observed response, likelihood-based criteria such as AIC and BIC can be compared.

Definition: Let \(Y\) be a continuous response and let \(Z=g(Y)\), where \(g\) is a one-to-one differentiable transformation. If a model specifies the density of \(Z\), then the corresponding density of \(Y\) is $$ f_Y(y)=f_Z(g(y))\left|g'(y)\right|. $$ For independent observations \(y_1,\ldots,y_n\), the log likelihood on the original response scale is $$ \ell_Y=\ell_Z+\sum_{i=1}^{n}\log\left|g'(y_i)\right|. $$ The sum is the log-Jacobian adjustment.

This tutorial considers fixed, differentiable transformations of continuous responses. If the transformation is not one-to-one, a density calculation must account for all inverse branches; the simple formula above is not sufficient. For discrete responses, probabilities rather than continuous densities are used, so there is no continuous Jacobian factor.

Deriving the Adjustment

The formula follows by requiring the probability assigned to a small interval to be the same before and after transformation. For one observation, a small change \(dy\) produces a change \(dz\) of approximately \(|g'(y)|dy\). Thus the probability near \(y\) is both \(f_Y(y)dy\) and \(f_Z(g(y))|g'(y)|dy\). Equating these expressions gives the density formula.

1
Relate the small intervals.
For a differentiable transformation, the transformed interval has length approximately \(|dz|=|g'(y)||dy|\).
2
Equate probability.
The same event has probability \(f_Y(y)|dy|=f_Z(g(y))|dz|\).
3
Convert to a density.
Substitute the interval relation and divide by \(|dy|\), giving \(f_Y(y)=f_Z(g(y))|g'(y)|\).
4
Use independence.
For independent observations, multiply the individual densities. Taking logarithms turns the product of Jacobian factors into a sum.

For a fixed transformation, the Jacobian depends on the observed responses, not on the regression coefficients. It is therefore constant across models that use the same observations and the same transformation. It does not generally cancel when one candidate uses a different transformation.

Theorem: For a fixed differentiable transformation, use the log likelihood on the original response scale $$ \ell_Y(\widehat{\theta})=\ell_Z(\widehat{\theta})+\log J,\qquad \log J=\sum_{i=1}^{n}\log|g'(y_i)|. $$ Then calculate AIC as \(-2\ell_Y+2k\) or BIC as \(-2\ell_Y+k\log n\), using the parameter count \(k\) for the fitted model. The likelihood term must refer to the same observed response scale for every model being compared.

Here, \(k\) includes the parameters estimated in the likelihood, including a variance parameter in a Gaussian model. For a fixed transformation, the Jacobian adjustment does not change \(k\). If a transformation parameter is estimated, however, its likelihood contribution depends on that parameter and it must be included when maximizing the likelihood; the parameter must also be handled in the criterion’s parameter count.

Worked Example: Comparing a Response With Its Logarithm

Worked Example: Comparing a Response With Its Logarithm

A hypothetical analyst observes three positive response values \(y=(1,2,4)\). Two candidate models each use an intercept and a Gaussian variance, so \(k=2\). Model O is Gaussian on the original response scale. Model L is Gaussian on \(z=\log y\). Both are fitted by maximum likelihood. Compare their AIC values. Use natural logarithms.

Method. For a Gaussian model with \(n\) observations and residual sum of squares \(RSS\), the maximized log likelihood is $$ \widehat{\ell}=-\frac{n}{2}\left[\log(2\pi)+1+\log\left(\frac{RSS}{n}\right)\right]. $$ This uses the maximum-likelihood estimate of the variance, \(RSS/n\), not the unbiased estimate \(RSS/(n-p)\).

Step 1: Model O on the original scale. Its fitted intercept is the sample mean: $$ \bar y=\frac{1+2+4}{3}=\frac{7}{3}. $$ The residuals are \(1-7/3=-4/3\), \(2-7/3=-1/3\), and \(4-7/3=5/3\). Therefore, $$ RSS_O=\left(-\frac{4}{3}\right)^2+\left(-\frac{1}{3}\right)^2+\left(\frac{5}{3}\right)^2 =\frac{16+1+25}{9}=\frac{14}{3}. $$ Thus \(RSS_O/n=(14/3)/3=14/9\). Using \(\log(2\pi)=1.837877\) and \(\log(14/9)=0.441833\), $$ \widehat{\ell}_O=-\frac{3}{2}(1.837877+1+0.441833)=-4.919565. $$ Its AIC is \(-2(-4.919565)+2(2)=13.8391\).

Step 2: Model L on the transformed scale. The transformed observations are \(0,\log 2,\log 4\), or \(0,0.693147,1.386294\). Their mean is \(0.693147\). The residuals are \(-0.693147,0,0.693147\), so $$ RSS_L=2(0.693147)^2=2(0.480453)=0.960906. $$ Now \(RSS_L/n=0.320302\), and \(\log(0.320302)\approx-1.138491\). The transformed-scale log likelihood is $$ \widehat{\ell}_Z=-\frac{3}{2}(1.837877+1-1.138491)=-2.549079. $$ This is not yet a log likelihood for the original \(y\) observations.

Step 3: Add the log-Jacobian. For \(z=\log y\), \(g'(y)=1/y\). Hence $$ \log J=\sum_{i=1}^{3}\log(1/y_i) =-\log(1)-\log(2)-\log(4) =-\log(8)=-2.079442. $$ The comparable log likelihood is \(\widehat{\ell}_Y=-2.549079-2.079442=-4.628521\). Model L therefore has $$ AIC_L=-2(-4.628521)+2(2)=13.2570. $$

Check. The adjusted log likelihood for Model L is on the original response scale, just like Model O’s. Its AIC, \(13.2570\), is lower than Model O’s \(13.8391\). The common mistake is to use \(-2(-2.549079)+4=9.0982\) as Model L’s AIC and compare that directly with Model O. That omits the Jacobian and understates Model L’s AIC by \(2\log 8=4.1589\). The corrected calculation avoids comparing densities on different scales.

What Cancels and What Does Not

Suppose several models use the same transformed response \(z=g(y)\), the same observations, and the same fixed transformation. They all receive the same log-Jacobian adjustment. Their AIC values therefore all shift by the same amount, \(-2\log J\), and their BIC values all shift by the same amount, \(-2\log J\). The adjustment does not change their ranking within that group.

When candidate models use different transformations, their Jacobian terms generally differ. Each likelihood must first be converted to the density scale of the same observed response. This applies to both AIC and BIC: their complexity penalties differ, but both use the same maximized log likelihood.

Worked Example: Two Models Using the Same Log Transformation

For hypothetical positive observations \(y=(1,2,3,4)\), two fitted Gaussian regression models use \(z=\log y\). Both use all four cases. Model A has \(RSS_A=2\), \(k_A=2\); Model B has \(RSS_B=1\), \(k_B=3\). Compare their AIC values on the original response scale.

Method. First calculate each Gaussian log likelihood on the \(z\) scale, then apply the shared Jacobian. Here, $$ \log J=-\sum_{i=1}^{4}\log y_i=-\log(24)=-3.178054. $$

For Model A, \(RSS_A/n=2/4=0.5\), and \(\log(0.5)=-0.693147\). Thus $$ \widehat{\ell}_{Z,A}=-2[1.837877+1-0.693147]=-4.289460. $$ After the Jacobian adjustment, \(\widehat{\ell}_{Y,A}=-4.289460-3.178054=-7.467514\). Its AIC is \(-2(-7.467514)+2(2)=18.9350\).

For Model B, \(RSS_B/n=1/4=0.25\), and \(\log(0.25)=-1.386294\). Therefore, $$ \widehat{\ell}_{Z,B}=-2[1.837877+1-1.386294]=-2.903166. $$ After adjustment, \(\widehat{\ell}_{Y,B}=-2.903166-3.178054=-6.081220\). Its AIC is \(-2(-6.081220)+2(3)=18.1624\).

Check. Model B has the lower AIC, \(18.1624\) versus \(18.9350\). Without the shared adjustment, the AIC values would be \(12.5789\) and \(11.8063\), respectively. The ranking is unchanged because both models use the same transformation and observations; each corrected AIC is higher by \(2\log(24)=6.3561\). The trap is to assume this cancellation still applies when candidate transformations differ.

Worked Example: Changing the Units of the Response

Worked Example: Changing the Units of the Response

A hypothetical five-case analysis compares two Gaussian models. The first is fitted directly to \(y\), with \(RSS=0.08\) and \(k=2\). The second is fitted to \(z=100y\), with \(RSS_Z=500\) and \(k=2\). Put both likelihoods on the \(y\) scale and compare AIC. The scale factor represents a change of units, not a change in the underlying observations.

For \(z=100y\), the derivative is \(g'(y)=100\), so $$ \log J=\sum_{i=1}^{5}\log(100)=5\log(100)=23.025851. $$ The log likelihood on the \(y\) scale equals the log likelihood on the \(z\) scale plus \(23.025851\).

For the model fitted to \(z\), \(RSS_Z/n=500/5=100\). Since \(\log(100)=4.605170\), $$ \widehat{\ell}_Z=-\frac{5}{2}[1.837877+1+4.605170]=-18.607618. $$ The common-scale log likelihood is \(-18.607618+23.025851=4.418233\). Its AIC is \(-2(4.418233)+2(2)=-4.8365\).

For the model fitted directly to \(y\), \(RSS/n=0.08/5=0.016\) and \(\log(0.016)=-4.135167\). Hence $$ \widehat{\ell}_Y=-\frac{5}{2}[1.837877+1-4.135167]=3.243225, $$ and its AIC is \(-2(3.243225)+4=-2.4865\). The model fitted to \(z\) has the lower AIC after correction.

Check. The transformed-scale AIC before adjustment is \(37.2152+4=41.2152\). Converting it to the \(y\) scale subtracts \(2\log J=46.0517\), giving \(-4.8365\), as above. A large or negative AIC is not by itself an error; its absolute value depends on the density’s measurement scale. The correction makes the comparison meaningful on a common scale.

Conditions and Common Traps

A valid likelihood-criterion comparison requires that the candidate likelihoods refer to the same observed cases and to the same response scale. The earlier material on candidate models and a common sample addresses why changing the observations can invalidate a direct comparison; a Jacobian does not fix a sample-size mismatch. Also verify that each model’s distributional assumptions are appropriate on the scale where it is specified. For example, a Gaussian model for \(\log Y\) implies a positive response \(Y\) with a lognormal-type conditional distribution, not a Gaussian response on its original scale.

  • Omitting the Jacobian across transformations. Add \(\sum_i\log|g'(y_i)|\) to the transformed-scale log likelihood before calculating the criterion.
  • Using the wrong sign. The log likelihood for \(Y\) is \(\ell_Z+\log J\). Consequently, the AIC and BIC adjustment is \(-2\log J\).
  • Applying a correction twice. If software reports a likelihood already evaluated as a density for \(Y\), do not add the Jacobian again.
  • Assuming every transformation has a constant Jacobian. For \(z=\log y\), the adjustment depends on the observed \(y_i\). For \(z=100y\), it is \(n\log 100\).
  • Treating an estimated transformation as fixed. If a transformation parameter is estimated, its Jacobian can depend on that parameter and must be included in the likelihood optimization.
  • Comparing incomparable likelihoods because the penalty is familiar. AIC and BIC formulas do not remove the need for a common response scale.
Key takeaway: A likelihood is a density relative to a scale. When a continuous response is transformed, the change-of-variables Jacobian converts that density back to the original response scale. Apply the adjustment before comparing AIC or BIC across different transformations; for models using the same fixed transformation and observations, the shared adjustment cancels from their ranking.

Check Your Understanding

For each problem, identify whether the Jacobian is needed for the comparison and calculate the requested adjustment.

  1. A model uses \(z=\log y\) for observations \(y=(2,5)\). What is the log-Jacobian adjustment to its transformed-scale log likelihood? (Answer: \(-\log(10)\approx-2.3026\).)
  2. There are six observations and \(z=10y\). By how much does the log likelihood change when converted from the \(z\) scale to the \(y\) scale, and by how much does AIC change? (Answer: Log likelihood increases by \(6\log(10)\approx13.8155\); AIC decreases by \(12\log(10)\approx27.6310\).)
  3. For \(z=(y^2-1)/2\), two observations are \(y=(1,3)\). Find the log-Jacobian adjustment. (Answer: \(\log(3)\approx1.0986\).)
  4. Two models use the same four positive responses, the same fixed log transformation, and have transformed-scale AIC values of 15 and 17. Which model has lower AIC after conversion to the original response scale? (Answer: The model with transformed-scale AIC 15; both receive the same adjustment, so the ranking is unchanged.)
  5. Two candidate models use the same observations, but one is Gaussian on \(y\) and one is Gaussian on \(\log y\). The latter has transformed-scale log likelihood \(-6.0\), and the observed responses have product \(e^3\). What is its original-scale log likelihood? (Answer: \(-6.0-3=-9.0\), since the log-Jacobian for the log transformation is minus the log of the response product.)

Previous

Next

Back to Tutorials
Clinical Biostats