Additive and Proportional Residual Error for Emax Fits

Why the Residual-Error Model Matters

An Emax curve specifies how the mean response changes with concentration. It does not, by itself, say how individual observations vary around that curve. That second part is the residual-error model. It matters because observations with different expected variability should not necessarily have equal influence on parameter estimates. A variance model that does not match the data can distort both the apparent precision of observations and the fitted curve.

Let \(Y_i\) be the continuous response observed at concentration \(C_i\), and let \(\mu_i=f(C_i;\theta)\) be the model-predicted mean, where \(\theta\) contains the Emax mean-model parameters. For example, a baseline-relative direct Emax mean model can be written as \(\mu_i=E_0+E_{\max}C_i/(EC_{50}+C_i)\). The residual is \(e_i=Y_i-\mu_i\). It is an observed difference after fitting; the error is the unobserved random deviation assumed by the model.

Three Common Variance Structures

In an additive error model, variability is constant on the response scale. In a proportional error model, the standard deviation increases in proportion to the predicted response magnitude. A combined model allows both a constant variance component and a response-dependent component.

The combined formula assumes that \(\epsilon_i\) and \(\eta_i\) are independent, mean-zero normal terms. The additive standard deviation \(\sigma_{\text{add}}\) has the same units as the response; \(\sigma_{\text{prop}}\) is dimensionless. A proportional model is most naturally interpreted when predicted means are positive and the response scale has a meaningful zero. If predictions can be negative or a zero is arbitrary, proportional error on the raw response scale may not be scientifically appropriate.

These are assumptions about variation around the mean curve, not alternative Emax mean models. For instance, changing from additive to combined error does not add an Emax term or change the definition of \(EC_{50}\). It can, however, change the estimated mean-model parameters because the fitting procedure gives observations different relative weights.

Likelihood, Weighting, and the Mean-Parameter Estimates

For independent normal observations, the log likelihood is obtained by multiplying the observation densities and taking the logarithm. Apart from no omitted terms, it is

where \(\psi\) denotes the error-model parameter or parameters. This expression has two parts for each observation: a squared residual divided by its variance, and a log-variance penalty. When the variances are fixed, maximizing the likelihood is equivalent to minimizing a weighted sum of squared residuals, with weight \(1/V_i\). When variance parameters are estimated, the log-variance term is essential: making every variance very large reduces the residual penalty but is penalized by \(\log(V_i)\).

Because \(V_i\) can depend on \(\mu_i\), and \(\mu_i\) depends on \(\theta\), the mean parameters and error parameters are generally estimated jointly. Under proportional error, a large predicted mean implies a larger variance and therefore a smaller weight for the same squared residual. Under additive error, observations have equal variance and equal weight. Neither weighting rule is universally preferable: the relevant question is which variance assumption is credible for the response and supported by the observed residual pattern.

Comparing Error Models

A likelihood comparison is meaningful only when models use the same response observations and the same likelihood convention. A common summary is the Akaike information criterion, \(AIC=-2\ell_{\max}+2k\), where \(\ell_{\max}\) is the maximized log likelihood and \(k\) is the number of estimated parameters, including variance parameters. Lower AIC indicates a better trade-off between fit and parameter count among the candidates compared; it is not a probability that a model is true.

The additive and proportional models each have one variance parameter, while the combined model has two. The mean-model parameters are counted in all three. Using maximum likelihood is important for AIC comparisons: restricted maximum likelihood values are not directly comparable in the usual way when the fixed-effect mean structure differs. A likelihood-ratio test can sometimes compare nested models, but inference requires its regularity conditions. Here, setting one variance component to zero places the parameter on a boundary, so the usual reference chi-square distribution may not apply directly. AIC provides a practical comparison without claiming that an ordinary likelihood-ratio test is automatically valid.

How Error Assumptions Can Move an Emax Fit

When an error model changes observation weights, it can move the fitted mean curve, not merely its estimated standard error. This is especially relevant when low- and high-concentration observations have different response magnitudes. Under additive error, a residual of 2 response units has the same squared-residual contribution at either level if the variance is constant. Under proportional error, the same raw residual can be much more influential at a low predicted mean, where the model expects less variability.

Diagnostics, Limitations, and Common Pitfalls

A useful first check is a plot of residuals against fitted response, with concentration also considered because the Emax mean changes systematically with concentration. Roughly constant vertical spread supports additive error. A widening fan as fitted response increases is compatible with proportional error. Spread that widens at higher response but remains substantial near low response can support combined error. These patterns are clues rather than proofs; mean-model misspecification can also produce structured residuals.

Residuals that are mostly positive in one concentration region and mostly negative in another suggest the mean curve may be inadequate, rather than merely indicating the wrong variance model. A variance model cannot repair a systematically wrong Emax mean. Conversely, a fan-shaped spread does not by itself establish proportional error: a few influential observations, response transformations, or unmodeled subgroups may produce similar appearances. Check whether estimated standard deviations are plausible on the response scale, and whether standardized residuals have a more stable spread under the proposed model.

The normality and conditional-independence assumptions also matter to the likelihood above. Strong skewness, outliers, repeated measurements with within-subject correlation, or censoring may require a different observation model. The three structures described here address heteroscedasticity; they do not solve dependence, response-scale problems, or an unsuitable mean curve. A combined model is more flexible, but its additional variance parameter can be weakly estimated when the data do not cover a broad response range.

• Pitfall: “The high-concentration residuals are larger, so the Emax mean model must be wrong.” Correction: larger raw residuals may be expected if variance increases with response. Assess spread relative to the variance predicted by the candidate error model, while separately checking for directional residual patterns.
• Pitfall: “The smallest weighted sum of squares identifies the best error model.” Correction: a likelihood for estimated variances includes both the scaled squared residuals and the log-variance terms. Compare maximized likelihoods or an appropriate information criterion, not weighted sums alone.
• Pitfall: “A proportional error model is suitable for any response because it is relative.” Correction: proportional error depends on the response zero and usually assumes positive predicted means. Check that this scale-based interpretation is scientifically meaningful.