Nonlinear Least Squares for Direct-Effect Models

From a Concentration-Effect Model to an Estimation Problem

A direct-effect model gives a predicted response at each measured concentration. Estimation chooses parameter values so those predictions are close to the observed responses. For a linear model, the least-squares solution can be calculated directly. For Emax and sigmoid Emax models, parameters such as \(EC_{50}\) enter nonlinearly, so fitting generally requires an iterative numerical search.

Let \(C_i\) be the concentration and \(y_i\) the observed effect for observation \(i\), where \(i=1,\ldots,n\). Write the model prediction as \(f(C_i;\boldsymbol{\theta})\), where \(\boldsymbol{\theta}\) is the parameter vector. The residual is the observed minus predicted effect. Ordinary, unweighted nonlinear least squares chooses parameters to minimize the sum of squared residuals.

For illustration, a stimulatory linear model can be written \(f(C)=E_0+sC\), with baseline \(E_0\) and slope \(s\). The Emax model is \(f(C)=E_0+E_{\max}C/(EC_{50}+C)\), and the sigmoid Emax model is \(f(C)=E_0+E_{\max}C^\gamma/(EC_{50}^\gamma+C^\gamma)\). Here \(E_{\max}\) is the maximum effect above baseline, \(EC_{50}\) is the concentration producing half of that maximum in the Emax model, and \(\gamma\) is the Hill coefficient. The same estimation principle applies to inhibitory parameterizations: use the specified model prediction and calculate its residuals.

The unweighted objective gives every observation the same squared-error weight. Its common statistical interpretation assumes errors are independent, have mean zero, and have constant variance; normally distributed errors additionally make least squares the maximum-likelihood method under that error model. The numerical minimization itself can still be defined without assuming normality. The concentration and response values must be paired correctly, and the chosen model must be meaningful for the observed range. These assumptions do not establish that a concentration causes an effect.

Linear Least Squares as a Reference Point

The linear concentration-effect model is linear in its parameters. Its RSS is a quadratic function of \(E_0\) and \(s\), and minimizing it gives the familiar least-squares solution. This provides a useful contrast: the objective is still RSS for an Emax curve, but the nonlinear dependence on \(EC_{50}\) usually prevents a single closed-form solution for all parameters.

Why Emax Fits Require Iteration

In the Emax model, changing \(EC_{50}\) changes the prediction at every nonzero concentration in a nonlinear way. A numerical optimizer therefore starts from a parameter vector, calculates predictions and residuals, proposes a change to the parameters, and evaluates the new RSS. The starting-value tutorial covers how to choose plausible initial values; here the focus is what the fitting algorithm does with them.

The basic local update can be derived by approximating the model near the current parameter vector \(\boldsymbol{\theta}\). Let \(J\) be the Jacobian matrix, whose element \(J_{ij}\) is the derivative of prediction \(f(C_i;\boldsymbol{\theta})\) with respect to parameter \(\theta_j\). For a small proposed change \(\boldsymbol{\delta}\), approximate the predictions by \(f(\boldsymbol{\theta}+\boldsymbol{\delta})\approx f(\boldsymbol{\theta})+J\boldsymbol{\delta}\). The residual vector after the change is therefore approximately \(\mathbf{r}-J\boldsymbol{\delta}\). Choosing the change that minimizes its squared length gives the normal equations below.

This is the Gauss–Newton step when the system has a suitable solution. The derivation is a local linear approximation, not a guarantee that the proposed step will reduce the actual, nonlinear RSS. Implementations commonly use damping, such as a Levenberg–Marquardt step, or a line search: if a proposal increases RSS, the algorithm reduces or modifies the step and checks again. The scale of parameters matters because \(E_0\), \(E_{\max}\), and \(EC_{50}\) can have different units and numerical magnitudes; the earlier material on scaling and reparameterizing Emax models is relevant when selecting a stable representation.

For \(f(C)=E_0+E_{\max}C/(EC_{50}+C)\), the prediction derivatives include

These derivatives supply the Jacobian at the current parameter values. For the sigmoid model, let \(q=(C/EC_{50})^\gamma\), so \(f=E_0+E_{\max}q/(1+q)\). For \(C>0\), derivatives with respect to \(EC_{50}\) and \(\gamma\) are \(-E_{\max}\gamma q/[EC_{50}(1+q)^2]\) and \(E_{\max}q\ln(C/EC_{50})/(1+q)^2\), respectively. At \(C=0\), the concentration-dependent contribution is zero. The optimizer uses such local sensitivities to determine which parameter changes could reduce the residuals.

Assessing Convergence and Detecting Failure

An optimizer stopping is not, by itself, proof that it found a useful minimum. Convergence checks should assess the objective, the parameter changes, and whether the fit is numerically and scientifically credible. A practical stopping rule commonly combines a small relative change in RSS with small scaled parameter changes; tolerances depend on the software and the scale of the problem.

A monotonically decreasing RSS is encouraging when the algorithm is designed to accept only improving steps. However, the decrease can become tiny because the optimum has been reached, because the parameters are weakly identified, or because the numerical method is stuck. A nearly singular \(J^\mathsf{T}J\) can signal that different parameter changes produce very similar predictions. Earlier material on identifiability and confounding explains why the data may not distinguish \(EC_{50}\), \(E_{\max}\), or \(\gamma\) well even when an optimizer reports convergence.

Other warning signs include an RSS that becomes undefined, non-finite predictions, large oscillating updates, and materially different solutions from different starts. Poor scaling can make one parameter appear unchanged while another continues to move. A small residual sum alone is not enough: the number and placement of concentrations, the response scale, and the model assumptions all affect whether the fitted parameters are informative. Detailed residual diagnostics and uncertainty assessment are separate steps; convergence does not replace them.

Choosing and Interpreting the Fitting Approach

For the linear model, ordinary least squares has a direct solution and avoids iterative-search issues. For Emax and sigmoid Emax models, nonlinear least squares directly minimizes discrepancies on the response scale and accommodates the model’s nonlinear concentration-effect shape. The cost is dependence on starting values and numerical behavior. A fit that converges is a numerical result, not evidence by itself that the selected model is correct or that every parameter is well determined.

The objective here is unweighted RSS. It treats a one-unit residual as equally costly at low and high fitted responses. Weighted fitting changes that objective and is a distinct method; it should not be introduced by informally rescaling a few observations. Similarly, choices about additive or proportional residual error change the error model, not merely the optimization algorithm. Keep the objective and its assumptions explicit when interpreting a fitted curve.