Worked Example Fitting an Emax Curve

Fitting a Direct Emax Curve

A fitted concentration–response curve summarizes how the expected response changes with concentration. Estimating the curve is not just a matter of drawing a smooth line through observed values: the chosen model, the range of concentrations, the residual pattern, and the precision of the estimates all affect what the fit can support. This tutorial works through a simple direct Emax fit, from data and starting values to parameter uncertainty.

The example uses baseline-adjusted responses, so the response expected at concentration zero is fixed at zero. That assumption is specific to this example; it is not appropriate when the response has a nonzero baseline. The model also uses a Hill coefficient fixed at one, giving the standard Emax curve rather than a sigmoid Emax curve.

For an unweighted nonlinear least-squares fit, the usual working assumptions are that the errors have mean zero, are independent across observations, and have a common finite variance. The response scale and concentration units must be consistent throughout. If the variability changes with concentration or response, unweighted least squares may give inefficient estimates and misleading uncertainty; the earlier material on weighted fitting and residual-error models addresses those alternatives. Replicate observations are permitted, but they must be independent observations for the stated error assumption to hold.

Starting Values and a Small Dataset

Starting values are initial guesses supplied to the numerical optimizer, not estimates to report as results. A reasonable starting value for \(E_{\max}\) is near the largest observed response when the data approach a plateau. A reasonable \(EC_{50}\) is near the concentration at which responses are about half that plateau. These visual clues are weak when the data do not include concentrations near the plateau or the half-maximal response; in that case, several starting values may be needed to check whether the fit is stable.

Why the Least-Squares Fit Is Nonlinear

The model is nonlinear in \(EC_{50}\), so the parameters are estimated together by minimizing the sum of squared differences between observations and curve predictions. For observations \(Y_i\) at concentrations \(C_i\), this objective is

A useful local justification for the iterative fit comes from the derivatives of the predicted response:

These derivatives describe how a small parameter change alters the curve. Near current parameter values, the model can be approximated by a linear change in parameters: the prediction changes by approximately the first derivative times the change in each parameter. A least-squares iteration uses this local approximation to choose an update that reduces the residual sum of squares, then recalculates the derivatives at the updated values. Iteration stops when parameter changes and objective changes are sufficiently small. This is a numerical procedure, not a guarantee that every starting value will find the same minimum; convergence, parameter plausibility, and residual diagnostics still require review.

In the paired dataset above, the cancellation can also be expressed through these derivatives. At a fixed concentration \(C\), both observations have the same derivative with respect to each parameter, so the pair's gradient contribution is proportional to the sum of its residuals. The displayed residual pairs are only approximately centered because the observations and fitted values are rounded; therefore they do not establish an exact stationary point. Stationarity should be checked using unrounded data and fitted parameters, and a practical fit should additionally verify that the optimizer has converged to a minimum and that the fitted values and residuals are sensible.

Residual Checks and Failure Modes

A residual is the observed response minus the fitted response, \(e_i=Y_i-\widehat{E}(C_i)\). Residuals should be examined against concentration and fitted response, not only summarized by the sum of squares. A plot can reveal curvature left over from the chosen model, a changing spread, an influential observation, or a cluster of residuals with the same sign. Those patterns can indicate model misspecification, unequal error variance, dependence, or a problematic data point; they should not be dismissed merely because the optimizer reports convergence.

In the worked dataset the residuals are paired approximately around zero at each concentration, with magnitudes about 1 response unit at the first three concentrations and 1.49 and 1.40 response units at 24. This constructed pattern is compatible with a constant-variance error model, but four concentration levels are too few to establish that assumption reliably. The pattern also does not prove the Emax model is biologically correct. It only shows that these data do not display an obvious systematic departure from this particular fitted curve.

The estimate is especially vulnerable when the observed concentrations fail to cover the curve’s informative regions. If all concentrations are far below \(EC_{50}\), then \(E(C)\) is approximately \(E_{\max}C/EC_{50}\): the data may identify the ratio more clearly than either parameter separately. If the observations have not approached a plateau, the estimated \(E_{\max}\) can be strongly influenced by the assumed curve shape and may have large uncertainty. If the concentrations are all far above \(EC_{50}\), the response may look nearly flat and reveal little about \(EC_{50}\). These are identification limitations, not problems that can be solved simply by adding decimal places to the estimates.

The main alternative fitting choices depend on the data and question. Weighted least squares is useful when observations have known or estimated differences in precision; a nonlinear likelihood may be preferable when the response distribution calls for a specific error model. A linear model may be a useful approximation over a narrow concentration range, but it does not represent a saturating plateau. A sigmoid Emax curve adds a Hill coefficient and can represent a steeper or shallower transition, but that extra flexibility can be poorly supported by sparse data. The earlier material on comparing linear and Emax models and choosing between Emax and sigmoid Emax motivates those choices; here the focus remains on fitting and checking the standard Emax curve.

Uncertainty in the Plateau and \(EC_{50}\)

A point estimate does not show how precisely the data determine a parameter. Under a well-behaved fit, an approximate covariance matrix can be obtained from the local derivatives and the estimated residual variance. In simple unweighted nonlinear least squares, the local approximation is \(s^2(J^\mathsf{T}J)^{-1}\), where \(J\) is the matrix of derivatives evaluated at the fitted parameters and \(s^2=SSE/(n-k)\). Here \(k\) is the number of fitted parameters; for the zero-baseline model, \(k=2\). The approximation is local and relies on the model and error assumptions being adequate.

From Fitted Parameters to a Response Prediction

Common Mistake and Practical Summary

Common mistake: “The optimizer converged, so the plateau and \(EC_{50}\) are well estimated.” Convergence only means the numerical procedure met its stopping criteria. It does not establish that the data cover the plateau and transition regions, that the residual assumptions are reasonable, or that alternative starting values would lead to the same solution. The correction is to inspect the fitted curve, residuals, parameter constraints, uncertainty intervals, and sensitivity to starting values before interpreting the estimates.

A standard Emax fit is most useful when its interpretation follows the data’s information content. Report the response and concentration units; distinguish the plateau from the half-maximal concentration; and describe uncertainty and limitations rather than presenting point estimates as exact biological constants. If the estimated plateau is poorly determined because the data do not approach saturation, that limitation should remain explicit even when the fitted curve looks smooth.