Baseline and Emax Confounding in Direct Models

Why Baseline and Emax Can Be Hard to Separate

A direct Emax curve has two distinct vertical features: its response at zero concentration and the change from that baseline toward the high-concentration limit. If observations do not adequately represent the baseline or the plateau, both features may be weakly constrained. A higher estimated baseline can then be offset by a smaller estimated maximal change, or vice versa, while predictions over the observed concentration range remain similar.

This is a parameter confounding problem, not necessarily evidence that the Emax model is wrong. It matters because the fitted curve may describe observed responses adequately while giving unstable estimates of baseline or maximum effect. The issue can be examined by holding EC50 and, where relevant, the Hill coefficient fixed, so that attention is on the relationship between the baseline and effect amplitude.

Definition: For an increasing direct Emax model, write the mean response at concentration $C$ as $$ \mu(C)=E_0+E_{\max}f(C), \qquad f(C)=\frac{C}{EC_{50}+C}. $$ Here $E_0$ is the mean response at $C=0$, and $E_{\max}$ is the increase from baseline to the asymptotic response $E_0+E_{\max}$. Thus $E_{\max}$ is not the absolute plateau response. For a sigmoid Emax model, $f(C)$ can instead be $C^\gamma/(EC_{50}^\gamma+C^\gamma)$; the baseline-amplitude relationship discussed here is the same when the other curve parameters are held fixed.

The model is a mean-response relationship. In a simple least-squares setting, observed responses can be written $Y_i=\mu(C_i)+\varepsilon_i$, where the errors have mean zero, are independent, and have constant variance. These assumptions are used for the least-squares calculations below; they are not required to recognize the underlying geometric issue in the mean curve. For an inhibitory response, the signed change may be negative, or a positive magnitude may be used with an explicit subtraction. The interpretation of baseline and amplitude should remain clear.

Why the Parameters Compensate

At each observed concentration, the response depends on $E_0$ and $E_{\max}$ through the two quantities $1$ and $f(C)$. If $f(C)$ barely varies across observations, increasing $E_0$ and decreasing $E_{\max}$ can offset one another. In the limiting case where every observation has the same effect fraction, only one combination of the two parameters is identified.

This can be seen by writing the model for observations $i=1,\ldots,n$ as a linear regression with predictor $f_i=f(C_i)$. The design matrix has columns of ones and $f_i$. Its determinant of cross-products is

\det(X^\mathsf{T}X) =n\sum_{i=1}^n(f_i-\bar f)^2.

To verify the expression, the cross-product matrix is

X^\mathsf{T}X= \begin{pmatrix} n & \sum_i f_i\\ \sum_i f_i & \sum_i f_i^2 \end{pmatrix}, \qquad \det(X^\mathsf{T}X) =n\sum_i f_i^2-\left(\sum_i f_i\right)^2 =n\sum_i(f_i-\bar f)^2.

If the effect fractions are all equal, the determinant is zero: the columns are proportional, so the data cannot separately determine the intercept and amplitude. If the fractions are merely close together, the determinant is small and estimates can be highly sensitive to noise. Under the independent, equal-variance error assumptions, the least-squares covariance matrix is proportional to $(X^\mathsf{T}X)^{-1}$. Its off-diagonal element is negative, reflecting the typical trade-off: higher baseline estimates accompany lower amplitude estimates.

Worked Example: A Nearly Flat Trade-Off in the Observed Range

Suppose EC50 is fixed at a known value, and the observed concentrations correspond to effect fractions $f=0.4,\ 0.5,\ 0.6$. Hypothetical responses are $18.1,\ 19.9,\ 22.1$ response units. Compare $E_0=10,\ E_{\max}=20$ with the alternative $E_0=10.5,\ E_{\max}=19$.

For the first parameter pair, predictions are $10+20(0.4)=18$, $10+20(0.5)=20$, and $10+20(0.6)=22$. With residual defined as prediction minus observation, the residuals are $-0.1,\ 0.1,\ -0.1$, so the squared residual sum is $$ (-0.1)^2+(0.1)^2+(-0.1)^2=0.0300 $$ response-units squared.

For the alternative pair, predictions are $10.5+19(0.4)=18.1$, $10.5+19(0.5)=20.0$, and $10.5+19(0.6)=21.9$. The residuals are $0,\ 0.1,\ -0.2$, giving $$ 0^2+(0.1)^2+(-0.2)^2=0.0500 $$ response-units squared. The alternative is not the least-squares fit to these data, but its residual sum is still small. Across this narrow range, a half-unit rise in baseline and a one-unit fall in amplitude change predictions by only $0.1,\ 0,\ -0.1$ response units. Thus the data distinguish the two parameter pairs less sharply than their parameter values alone might suggest.

Assessing the Trade-Off in a Fit

A useful assessment asks whether materially different baseline-amplitude pairs produce similar fitted responses over the concentrations actually observed. The following calculation uses an idealized baseline observation, a response at $f=0.5$, and a plateau-limit response at $f=1$. The plateau point represents the asymptotic limit; at any finite concentration the Emax fraction is less than one.

Worked Example: Checking an Alternative Parameter Pair

Consider hypothetical responses $25/12,\ 10/3,\ 61/12$ response units at effect fractions $f=0,\ 0.5,\ 1$. Fit $E_0+E_{\max}f$ by least squares, with the effect fractions treated as fixed.

The mean of the three effect fractions is $0.5$, and the mean response is $3.5$. The least-squares slope is $$ \widehat E_{\max} =\frac{\sum_i(f_i-\bar f)(Y_i-\bar Y)} {\sum_i(f_i-\bar f)^2} =\frac{(-0.5)(-17/12)+(0)(-1/6)+(0.5)(19/12)} {0.25+0+0.25} =\frac{1.5}{0.5}=3. $$

The intercept is $\widehat E_0=\bar Y-\widehat E_{\max}\bar f=3.5-3(0.5)=2$. The fitted responses are therefore $2,\ 3.5,\ 5$. The residuals are $-1/12,\ 1/6,\ -1/12$, and the least-squares residual sum of squares is $$ \left(-\frac{1}{12}\right)^2+\left(\frac{1}{6}\right)^2+ \left(-\frac{1}{12}\right)^2 =\frac{1}{24}\approx 0.04167 $$ response-units squared.

Now use the alternative $E_0=2+1/17$ and $E_{\max}=3-2/17$. Its prediction changes from the fitted value by $1/17,\ 0,\ -1/17$ at the three effect fractions. Its residuals are consequently $$ -\frac{1}{12}+\frac{1}{17}=-\frac{5}{204},\qquad \frac{1}{6},\qquad -\frac{1}{12}-\frac{1}{17}=-\frac{29}{204}. $$ The alternative squared residual sum is $$ \left(-\frac{5}{204}\right)^2+\left(\frac{1}{6}\right)^2+ \left(-\frac{29}{204}\right)^2 =\frac{25}{41616}+\frac{1}{36}+\frac{841}{41616} \approx 0.04859 $$ response-units squared. This is an increase of $2/289$ from $1/24$. The example shows how a noticeable parameter trade-off can leave fitted responses relatively similar, while also showing that the alternative is not an equally good fit: its residual sum is larger.

In an applied fit, inspect the observed concentration coverage and the uncertainty in both parameters, not just the residual sum or a single plotted curve. A useful diagnostic is to compare fitted predictions with those from other plausible baseline-amplitude pairs. If predictions in the observed range barely change while the estimated baseline or plateau changes substantially, the data provide weak separation of those quantities. A strong negative correlation between parameter estimates is another warning sign, although its size depends on the model, design, and error assumptions.

The response scale also matters. Baseline and amplitude must be interpreted in the response units used in the analysis. If the response baseline was measured with substantial error, fixing it at a noisy observed value can create a misleading appearance of precision in $E_{\max}$. Conversely, if no baseline information is available, a fitted intercept may be an extrapolation rather than a directly supported estimate.

What Helps Separate Baseline from Amplitude?

Baseline observations near $C=0$ directly inform $E_0$, since $f(0)=0$. High concentrations that approach the plateau inform $E_0+E_{\max}$. When both regions are represented, the baseline and amplitude can be inferred from distinct parts of the curve rather than relying on a narrow middle range. In practice, a finite high concentration only approximates the plateau, and the adequacy of that approximation depends on its effect fraction.

Worked Example: Using Baseline and Plateau Information

Suppose hypothetical mean responses are observed at effect fractions $0,\ 0.5,\ 1$, with values $10,\ 20,\ 30$ response units, respectively. Under the stated model, the baseline observation gives $$ E_0=10. $$

The plateau-limit response is $E_0+E_{\max}=30$, so the amplitude is $$ E_{\max}=30-10=20 $$ response units. The midpoint prediction is $10+20(0.5)=20$, matching the middle observation. The endpoint observations directly anchor baseline and plateau, while the middle observation is a consistency check on the specified curve shape. With noisy data, estimates will not generally equal individual observations exactly, but coverage of low and high effect fractions still supplies distinct information about the two parameters.

Designing for separation does not mean collecting only baseline and very high-concentration observations. Intermediate concentrations remain important for assessing the curve between its limits and for estimating EC50 and, in a sigmoid model, the Hill coefficient. But if all observed responses lie in a narrow intermediate band, baseline-amplitude confounding can persist even when there are many observations. More observations at nearly the same effect fraction reduce random noise there; they do not replace coverage of distinct portions of the curve.

Common Mistake, Alternatives, and Limits

Common mistake: “The fitted curve passes close to the data, so $E_0$ and $E_{\max}$ must each be well estimated.” A good fit in the sampled range does not establish that conclusion. The correction is to assess parameter uncertainty and whether different parameter pairs give similar predictions over the observed range. A curve can be well described locally while its baseline or plateau remains poorly determined.

A direct way to reduce this confounding is to improve design coverage by measuring responses near baseline and near the plateau, where feasible. Another possibility is to use a baseline measured independently, but treating it as known is justified only when its measurement uncertainty is negligible for the purpose at hand. Fixing a parameter based on convenience rather than reliable information can understate uncertainty. Relative-to-baseline response definitions can make the baseline explicit in the data scale, but they do not create information about an unobserved plateau.

These checks do not resolve every source of uncertainty. If EC50 or the Hill coefficient is also poorly determined, changes in curve shape may compound the baseline-amplitude trade-off; the examples here held those features fixed to isolate one relationship. Model misspecification, time-varying responses, and an inappropriate residual-error model can also affect interpretation. Parameter confounding is therefore a warning about what the design supports, not by itself a diagnosis of which model or error assumption is wrong.

Key takeaway: In a direct Emax model, $E_0$ sets the response at zero concentration and $E_{\max}$ sets the change from baseline to the asymptote. When observed effect fractions are narrowly distributed, these parameters can compensate. Baseline and near-plateau information, together with uncertainty and prediction checks, help assess whether they are separately supported.

Check Your Understanding

Answer each question, then compare with the brief solutions below.

In $\mu(C)=E_0+E_{\max}C/(EC_{50}+C)$, what is the asymptotic response as concentration increases?
Why does observing only one effect fraction fail to identify both $E_0$ and $E_{\max}$?
If $E_0$ rises by 1 response unit and $E_{\max}$ falls by 2 response units, what is the prediction change at $f=0.5$?
What information does an observation at zero concentration provide in this model?
Why can a small residual sum of squares coexist with weakly determined baseline and amplitude estimates?

Solutions. 1. The limit is $E_0+E_{\max}$. 2. At a single fraction $f$, the response depends only on the combination $E_0+fE_{\max}$, so infinitely many parameter pairs give that same mean. 3. The change is $1+0.5(-2)=0$ response units. 4. Since $f(0)=0$, the mean response there is $E_0$. 5. If observed effect fractions cover only a narrow range, compensating parameter pairs can produce similar predictions in that range even though their baseline and plateau differ.