Smearing Estimates for Arithmetic-Mean Predictions

Why Exponentiation Is Not Enough

Suppose a positive outcome $Y$, measured in its original units, is modeled after taking its natural logarithm. A fitted value on the log scale can be exponentiated to return to the outcome scale. But exponentiating a fitted log-scale mean does not, in general, give the arithmetic mean of $Y$. The gap arises because the outcome also varies around its fitted log-scale value, and exponentiation is nonlinear.

For an arithmetic-mean prediction, the relevant question is how the log-scale residuals behave. Smearing estimates the amount by which exponentiating a fitted log-scale value must be adjusted to account for that residual variation. It uses the observed residuals rather than requiring the residuals to follow a particular named distribution.

Definition: Let $Y>0$, let $x$ denote the predictor values, and suppose the log-outcome model is $$ \log(Y)=\eta(x)+\varepsilon, $$ where $\eta(x)=E(\log(Y)\mid x)$ and $E(\varepsilon\mid x)=0$. The conditional smearing factor at $x$ is $$ S(x)=E\bigl(\exp(\varepsilon)\mid x\bigr). $$ The arithmetic conditional mean is $$ E(Y\mid x)=\exp\bigl(\eta(x)\bigr)S(x). $$

The factor $S(x)$ is dimensionless: it adjusts the exponentiated log-scale prediction without changing the outcome's units. It is at least 1 under the stated zero-conditional-mean assumption, because Jensen's inequality gives $E(\exp(\varepsilon)\mid x)\geq \exp(E(\varepsilon\mid x))=1$. This does not mean every individual fitted prediction must be raised by the same amount; the appropriate factor may depend on $x$.

In practice, $\eta(x)$ is estimated by fitting a regression to $\log(Y)$. Write the fitted log value for observation $i$ as $\widehat{\eta}(x_i)$, and define its log-scale residual as $e_i=\log(y_i)-\widehat{\eta}(x_i)$. A pooled smearing estimate is the average of the exponentiated residuals:

\widehat{S}=\frac{1}{n}\sum_{i=1}^{n}\exp(e_i), \qquad \widehat{E}(Y\mid x_0)=\exp\bigl(\widehat{\eta}(x_0)\bigr)\widehat{S}.

Here $x_0$ is a predictor value at which a prediction is wanted. The pooled estimate treats the observed exponentiated residuals as representative of the residual variation relevant at $x_0$. That is a substantive modeling assumption, not an automatic consequence of fitting a log-outcome regression.

Calculating a Pooled Smearing Estimate

Worked Example: Correcting One Log-Scale Prediction

Consider a hypothetical regression for a positive outcome measured in hours. At a particular predictor value $x_0$, the fitted log-scale value is $\widehat{\eta}(x_0)=2.00$. Four hypothetical fitted observations have log-scale residuals $-0.20,-0.10,0.10,$ and $0.20$. Calculate the pooled smearing estimate and the arithmetic-mean prediction at $x_0$.

First exponentiate each residual and average:

\widehat{S} =\frac{\exp(-0.20)+\exp(-0.10)+\exp(0.10)+\exp(0.20)}{4} =\frac{0.8187+0.9048+1.1052+1.2214}{4} =\frac{4.0501}{4} =1.0125.

The uncorrected exponentiated fitted value is $\exp(2.00)=7.389$ hours. Applying the estimated smearing factor gives

\widehat{E}(Y\mid x_0)=\exp(2.00)(1.0125) =7.389(1.0125) =7.481\text{ hours}.

For these hypothetical data, the arithmetic-mean prediction is about 7.481 hours. It is slightly larger than the exponentiated log-scale fitted value, 7.389 hours, because the residuals vary around zero and exponentiation gives greater weight to positive deviations. The calculation assumes this small residual sample represents the conditional residual distribution at $x_0$; with only four residuals, the estimate would be imprecise.

The average is taken after exponentiating each residual. Averaging residuals first and then exponentiating would not make the correction: under an intercept-containing least-squares fit, residuals may average to zero, which would produce $\exp(0)=1$ and discard the variation the correction is meant to capture.

Why the Residual Correction Works

The central result follows by substituting the log-outcome model into $Y$. Since $Y=\exp(\eta(x)+\varepsilon)$, exponent rules give $Y=\exp(\eta(x))\exp(\varepsilon)$. At a fixed $x$, $\eta(x)$ is constant, so it can be taken outside the conditional expectation:

\begin{aligned} E(Y\mid x) &=E\bigl(\exp(\eta(x)+\varepsilon)\mid x\bigr)\\ &=\exp(\eta(x))E\bigl(\exp(\varepsilon)\mid x\bigr)\\ &=\exp(\eta(x))S(x). \end{aligned}

This identity does not require normally distributed errors. Smearing estimates the expectation of the exponentiated error directly, using $\exp(e_i)$ as an empirical counterpart to $\exp(\varepsilon_i)$. Replacing the unknown $\eta(x_0)$ by its fitted value and the unknown factor by an estimate yields the practical prediction formula.

The identity does require the log-scale model to describe the conditional log mean: $E(\log(Y)\mid x)=\eta(x)$. If the fitted function is systematically wrong, its residuals can contain a pattern in $x$, and the correction does not repair that misspecification. In addition, a single pooled factor estimates $S(x)$ adequately only when the conditional expectation of $\exp(\varepsilon)$ is approximately the same across the predictor values to which that factor is applied.

When a Common Factor Can Mislead

A common factor is most plausible when the conditional residual distribution is stable across predictor values. Equal residual variances alone are not a complete guarantee: different distributional shapes can have different averages of $\exp(\varepsilon)$. Conversely, residual distributions can differ in other respects while yielding similar smearing factors. The quantity directly relevant to the arithmetic mean is $E(\exp(\varepsilon)\mid x)$.

Worked Example: Different Factors Across Two Predictor Groups

Suppose a hypothetical log-outcome model has two predictor groups, A and B. The fitted model has residuals $-0.40$ and $0.40$ in group A, and $-0.10$ and $0.10$ in group B. Each group has two observations. Estimate a smearing factor for each group and compare a prediction for group B using its own factor with one using the pooled factor.

For group A, the factor estimate is

\widehat{S}_A =\frac{\exp(-0.40)+\exp(0.40)}{2} =\frac{0.6703+1.4918}{2} =1.0811.

For group B, it is

\widehat{S}_B =\frac{\exp(-0.10)+\exp(0.10)}{2} =\frac{0.9048+1.1052}{2} =1.0050.

Because the groups have equal sample sizes, the pooled factor is the average of the four exponentiated residuals:

\widehat{S} =\frac{1.0811+1.0050}{2} =1.0430.

Suppose the fitted log value for a new group B case is 1.50, so its exponentiated fitted value is $\exp(1.50)=4.482$ outcome units. Using the group B estimate gives $4.482(1.0050)=4.504$ units. Using the pooled estimate instead gives $4.482(1.0430)=4.675$ units. In this hypothetical calculation the pooled factor raises the group B prediction more, because it also reflects group A's larger residual spread. The group-specific estimates are themselves based on just two residuals each, so the numerical difference is illustrative rather than a reliable empirical conclusion.

This example shows why plotting residuals against predictors and checking for changing spread or shape matters even when the log-scale mean function appears reasonable. If residual behavior changes systematically, a single factor may average together unlike conditions. Possible responses include estimating factors within scientifically meaningful groups or modeling how the relevant residual distribution changes with predictors. Such choices need adequate data and should be justified rather than selected solely to obtain a preferred prediction.

A Parametric Alternative Under Normal Errors

If the conditional errors are normally distributed with mean zero and a constant variance, there is a familiar formula for the correction. Specifically, if $\varepsilon\mid x\sim N(0,\sigma^2)$ for every $x$, then $E(\exp(\varepsilon)\mid x)=\exp(\sigma^2/2)$. Thus the common correction can be estimated as $\exp(s_e^2/2)$, where $s_e^2$ estimates the log-scale error variance. This formula depends on the normal, constant-variance assumption; it is not a universal substitute for checking residual behavior.

Derivation: For $\varepsilon\sim N(0,\sigma^2)$, its moment-generating function is $E(\exp(t\varepsilon))=\exp(t^2\sigma^2/2)$. Substituting $t=1$ gives $E(\exp(\varepsilon))=\exp(\sigma^2/2)$, which is the normal-theory smearing factor.

Worked Example: Applying the Normal-Theory Correction

Suppose a hypothetical log-outcome regression has estimated log-scale residual standard deviation $s_e=0.40$, and the fitted log value at a predictor setting is 1.20. Assume for this calculation that errors are approximately normal, have mean zero, and have constant variance across predictor settings.

The estimated variance is $s_e^2=(0.40)^2=0.16$. Therefore the normal-theory factor is

\widehat{S}_{N} =\exp\left(\frac{s_e^2}{2}\right) =\exp\left(\frac{0.16}{2}\right) =\exp(0.08) =1.0833.

The exponentiated fitted value is $\exp(1.20)=3.320$ outcome units. The corrected arithmetic-mean prediction is

3.320(1.0833)=3.597\text{ outcome units}.

This result is conditional on the stated normal, constant-variance error model. If residuals are strongly skewed or their distribution changes with predictors, the normal-theory factor may not capture the relevant conditional average. A residual-based smearing estimate avoids assuming normality, but it still requires representative residuals and an appropriate factor for the prediction setting.

Assumptions, Limitations, and Common Mistakes

For a useful arithmetic-mean prediction, separate two questions. First, does the fitted log-scale regression adequately estimate the conditional mean of $\log(Y)$? Second, does the chosen smearing factor adequately estimate $E(\exp(\varepsilon)\mid x_0)$? A correct answer to the first does not guarantee a correct answer to the second.

Positive outcomes are required. The ordinary logarithm is defined here only for $Y>0$. A smearing factor does not make this log model suitable for zero or negative outcomes.
Residuals are estimates, not the true errors. The fitted residuals are calculated after estimating the regression. In finite samples, they can be constrained by the fitting process and may not reproduce the full error distribution exactly. The pooled average is therefore an estimate, not an exact finite-sample correction.
The prediction setting must be represented. A pooled factor may be inappropriate if the residual distribution, and hence $E(\exp(\varepsilon)\mid x)$, changes with predictors. Extrapolating a common factor to settings unlike those in the data is especially risky.
The sample average can be unstable. Exponentiation gives large positive residuals substantial influence. A small number of large residuals can make the estimated factor variable, particularly in a small sample.
Do not confuse targets. Exponentiating a fitted conditional log mean gives a log-scale-based quantity, not automatically the conditional arithmetic mean. The arithmetic mean requires the additional residual correction.

A common incorrect argument is: “The fitted residuals average to zero, so the average exponentiated residual is one and no correction is needed.” The correction is not $\exp(\text{average residual})$; it is the average of exponentiated residuals. For example, residuals $-0.20,-0.10,0.10,0.20$ average to zero, but their exponentials average to 1.0125, not 1. The arithmetic mean responds to the full residual distribution, not just its average.

Key takeaway: To estimate an arithmetic mean from a log-outcome regression, multiply the exponentiated fitted log mean by an estimate of the conditional mean of exponentiated residuals. A pooled smearing factor is defensible only when it represents that conditional quantity at the prediction setting.

Check Your Understanding

Use the definitions and calculations above to answer these questions.

A log-outcome model has fitted log value 1.4 and an estimated smearing factor of 1.08. Write the arithmetic-mean prediction formula and explain which quantity the factor adjusts.
Why is the average of exponentiated residuals generally not equal to the exponential of the average residuals?
State the condition on the conditional residual distribution that makes a single pooled smearing factor reasonable across predictor values.
What additional assumptions justify using $\exp(s_e^2/2)$ as a common correction factor?
Give one reason a residual-based smearing estimate may still be unreliable, even though it does not require normally distributed errors.