Why a Nonmonotonic Dose-Response Needs Careful Evaluation
A dose-response curve that rises and then falls can matter for dose selection: the highest tested dose may not produce the greatest estimated efficacy. But a lower observed response at one high dose does not, by itself, establish a real downturn. Sampling variability, influential observations, an unsuitable response model, or differences between dose groups can all create an apparent peak.
The relevant question is therefore not simply whether the largest observed mean occurs below the maximum tested dose. It is whether the data, under a credible analysis, support a peak or decline strongly enough to inform a dose decision. This requires distinguishing evidence about the shape of the curve from evidence that a particular dose should be selected.
The response direction must be stated first. In the examples below, a larger value means greater efficacy, and responses are expressed in hypothetical efficacy units. If a smaller value is better for the endpoint being analyzed, the interpretation of a peak or downturn must be reversed. The conclusions concern the modeled dose-response, not a causal mechanism for its shape.
Models and Assumptions
A monotonic model, such as a linear trend or an increasing Emax curve, can describe a response that rises and then approaches a plateau. A plateau means that additional dose brings little estimated efficacy gain; it does not predict that efficacy decreases. A nonmonotonic model allows a decline. One simple choice is a quadratic curve, which is useful for describing a broad rise and fall but should not be treated as a mechanistic law.
The model is only useful if its assumptions are reasonably defensible. For a simple comparison of dose-group means, assume dose assignment and conduct make the groups comparable, observations from different participants are independent, and each group mean estimates the same defined response at its assigned dose. In the numerical regression example below, the usual F test additionally assumes independent, approximately normal errors with a common variance across dose groups. If means are based on different numbers of participants or have markedly different precision, an equal-weight analysis may be inappropriate.
In clinical data, the interpretation also depends on how the response is defined, how missing outcomes are handled, and whether placebo response or other design features have been accounted for. A downturn should not be attributed to dose if dose groups differ in ways that could explain it. The model and its error assumptions should be specified before using a test or interval to support a dose decision.
Worked Example: Fitting a Quadratic Curve and Locating Its Peak
Worked Example: Fitting a Quadratic Curve and Locating Its Peak
Suppose a hypothetical study reports one equally precise mean efficacy response at each of five doses. Larger responses indicate greater efficacy. The responses are assumed independent with approximately normal errors and common variance for the illustrative regression comparison.
| Dose (mg) | Scaled dose \(t=(d-100)/50\) | Mean response (efficacy units) |
|---|---|---|
| 0 | -2 | 0 |
| 50 | -1 | 4 |
| 100 | 0 | 7 |
| 150 | 1 | 6 |
| 200 | 2 | 3 |
Fit the quadratic using the equivalent centered basis \(1,t,z\), where \(z=t^2-2\). For these five equally spaced values, \(z\) is orthogonal to both the intercept and \(t\): \(\sum z=0\) and \(\sum tz=0\). Thus the least-squares coefficient of \(z\) is \(\sum zy/\sum z^2\). The sums are $$ \sum y=20,\qquad \sum ty=8,\qquad \sum t^2=10,\qquad \sum zy=-18,\qquad \sum z^2=14. $$
Therefore the intercept is \(20/5=4\), the slope on \(t\) is \(8/10=0.800\), and the coefficient on \(z\) is \(-18/14=-9/7\). The fitted curve is $$ \widehat{E(Y\mid t)}=4+0.800t-\frac{9}{7}(t^2-2). $$
Differentiating with respect to \(t\) gives \(0.800-(18/7)t\). Setting this equal to zero gives \(t^*=14/45\approx0.3111\), so the estimated vertex is at $$ d^*=100+50(14/45)\approx115.6\text{ mg}. $$ The fitted vertex is within the tested range, and the quadratic coefficient is negative, so this model describes an interior maximum. The fitted response there is approximately \(6.696\) efficacy units. This is a model-based estimate, not proof that 115.6 mg is the optimal clinical dose.
For a conventional nested-model comparison, the linear fit is \(4+0.800t\). Its residuals are \(-2.4,0.8,3.0,1.2,-2.6\), giving \(SSE_{\text{linear}}=23.60\). Adding the quadratic term reduces the sum of squared errors by \((9/7)^2(14)=23.1429\), leaving \(SSE_{\text{quadratic}}=0.4571\). Thus $$ F=\frac{(23.60-0.4571)/1}{0.4571/2}\approx101.3. $$ With 1 and 2 degrees of freedom, the upper-tail p-value is approximately 0.0097. Under the stated error assumptions, this is evidence against the linear model in favor of the quadratic term. The very small residual degrees of freedom, however, make the comparison fragile; it does not validate the quadratic shape outside the tested doses or establish a dose-selection decision.
Evidence for a Downturn Is Not the Same as a Dose Decision
A downward pattern can be assessed in more than one way. A prespecified contrast comparing high and lower doses asks about that particular difference. A curvature term asks whether the pattern across several doses departs from a straight line. These questions are not identical: pooling information across dose levels can support curvature even when one pairwise comparison is imprecise.
Worked Example: A Pairwise Decline That Remains Uncertain
Consider hypothetical mean responses of 2.0, 4.0, 6.0, 5.5, and 4.8 efficacy units at doses 0, 50, 100, 150, and 200 mg, respectively. Suppose each mean has standard error 0.5 efficacy units, and the means are independent and approximately normal. The observed mean is highest at 100 mg, but evaluate whether the 200 mg response is credibly lower than the 100 mg response.
The estimated difference, defined as the 100 mg mean minus the 200 mg mean, is \(6.0-4.8=1.2\) efficacy units. Under independence, its standard error is $$ SE=\sqrt{0.5^2+0.5^2}=0.7071\text{ efficacy units}. $$ A normal-approximation 95% confidence interval is $$ 1.2\pm1.96(0.7071)=1.2\pm1.386=(-0.186,\ 2.586)\text{ efficacy units}. $$ The interval includes zero. At the nominal 5% level, these data do not establish that the mean at 100 mg exceeds the mean at 200 mg. The point estimates show a decline, but its size is uncertain. On this pairwise comparison alone, the data do not justify selecting a dose below 200 mg on the basis of a confirmed efficacy downturn.
The converse distinction also matters: evidence of curvature does not pinpoint the best dose precisely. A fitted vertex can be unstable when the curve is flat near its maximum, when observations are sparse around the peak, or when uncertainty in the slope and curvature is substantial. Dose selection should consider the estimated response across a useful range, not just the numerical location of a fitted vertex.
Worked Example: Curvature Evidence and Pairwise Uncertainty Can Coexist
Use the five hypothetical means in the preceding example, and suppose their standard errors are known to be 0.5 efficacy units with independent normal sampling errors. For \(t=-2,-1,0,1,2\), define \(z=t^2-2\), whose values are \(2,-1,-2,-1,2\). The estimated quadratic coefficient in the centered basis \(1,t,z\) is $$ \hat{\beta}_2=\frac{\sum zy}{\sum z^2} =\frac{(2)(2)+(-1)(4)+(-2)(6)+(-1)(5.5)+(2)(4.8)}{14} =\frac{-7.9}{14}=-0.5643. $$
Because the means are independent with standard error 0.5, the standard error of this coefficient is \(0.5/\sqrt{14}=0.1336\) efficacy units per \(t^2\). The standardized statistic is \(-0.5643/0.1336=-4.22\), which gives a two-sided normal-approximation p-value of about 0.000024 for a zero quadratic coefficient.
The fitted linear coefficient is \(\hat{\beta}_1=\sum ty/\sum t^2=7.1/10=0.710\). The fitted vertex is therefore at \(t=-0.710/[2(-0.5643)]\approx0.629\), or \(d\approx131.5\) mg. The negative curvature gives evidence of a concave pattern under the assumptions, while the separate 100-versus-200 mg interval still includes zero. There is no contradiction: the curvature calculation uses the pattern across all five means, whereas the interval addresses one pairwise difference. Neither calculation alone establishes that 131.5 mg is the correct dose to select.
Model Comparison, Diagnostics, and Failure Modes
The model comparison should reflect the scientific question and the data available. A monotonic Emax model can represent increasing efficacy approaching a plateau. A quadratic can represent a broad rise and fall, but may behave implausibly beyond the observed dose range. More flexible curves can follow complex patterns, but flexibility also makes them easier to fit to noise. No model-selection statistic can replace checking whether the estimated shape is plausible and supported by observations near the apparent peak.
| Approach | What it can answer | Main caution |
|---|---|---|
| Monotonic Emax or trend model | Whether response rises and may approach a plateau | Cannot represent a genuine decline within its monotonic range |
| Quadratic model | Whether a simple curved pattern has an interior peak | May impose an unrealistic shape, especially outside observed doses |
| Prespecified dose contrast | Whether a particular pair of dose means differs | Does not describe the full dose-response shape |
| Flexible or mechanistic model | Whether a richer shape is supported by appropriate data and rationale | Can be weakly identified or overfit with sparse dose coverage |
Inspect the data and fitted curves together. In particular, check whether the apparent decline is driven by a single dose group, whether there are enough dose levels on both sides of the proposed peak, whether uncertainty is larger at high doses, and whether residuals show systematic patterns. Compare observed means and uncertainty with model predictions; a smooth fitted curve can hide poor fit. Examine sensitivity to reasonable alternative models and to influential observations. If the estimated peak is at a boundary, the data have not located an interior maximum.
Model comparisons also require care. A nested-model F test is appropriate only when its error assumptions are credible. Comparing several candidate curves and reporting only the one that produces the most attractive peak can exaggerate evidence, particularly when model choice was data-driven. A non-nested model comparison may use likelihood-based criteria, but relative fit does not establish that either model is an adequate description. Sparse dose-ranging data commonly leave the peak poorly determined.
Why This Matters for Dose Selection
A supported downturn can be relevant because a dose below the maximum tested dose may offer similar or greater estimated efficacy. But efficacy shape is one part of the decision, not an automatic selection rule. The strength of evidence, precision of the estimated curve, dose coverage, and robustness to model choice all matter. A conclusion should distinguish what the data show from what remains uncertain.
A common mistake is: “The observed mean at a lower dose is largest, so the lower dose is the efficacy optimum.” That reasoning treats noisy sample means as the true dose-response and ignores uncertainty. The correction is to estimate the response pattern under a stated model, assess the evidence for a decline and the uncertainty around the peak, and check that the conclusion is robust to plausible alternatives. A second mistake is to interpret a statistically significant quadratic term as proof of a clinically useful peak. It is evidence about curvature under the model assumptions, not proof that any particular lower dose should be chosen.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A quadratic fit has a negative curvature coefficient, but its vertex lies above the highest tested dose. Does it demonstrate an interior peak in the studied range?
- For a dose-response with \(t=-1,0,1\) and fitted curve \(5+0.6t-0.3t^2\), calculate the vertex and say whether it is a maximum.
- A 100 mg mean exceeds a 200 mg mean, but the confidence interval for their difference includes zero. What conclusion is justified about a downturn?
- Why can a curvature test be significant while a particular pairwise comparison of two dose means is not?
Solutions: (1) No. A negative curvature coefficient implies a maximum for the quadratic, but if its vertex is outside the studied range, it does not establish an interior peak among tested doses. (2) The vertex is \(t^*=-0.6/[2(-0.3)]=1\). Since the coefficient of \(t^2\) is negative, this is a maximum; whether it is within the studied dose range depends on that range. (3) The observed means decline, but the comparison does not establish a reliable decline at the confidence level used; do not claim the lower dose is proven superior. (4) A curvature test uses the pattern across several dose levels, while a pairwise comparison uses only two means. They test different features and can have different precision.