Diagnosing Nonmonotonic Dose-Response Patterns

Why a Dose-Response Curve May Turn

Dose selection often starts with a simple expectation: increasing dose increases efficacy, or increases the probability of a safety event. Yet biological responses need not follow a monotone pattern over every dose range. Efficacy may rise and then fall, while a safety endpoint may decline at first and increase later. Such patterns can be scientifically plausible, but a few irregular sample means do not establish them. The central task is to decide whether the observed shape is supported by the data, or could instead reflect sampling variation, an unsuitable model, or inadequate dose coverage.

A diagnosis matters because a monotone model can hide a meaningful downturn, while a flexible curve can mistake random variation for a peak. Either error can distort dose comparisons. A credible pattern should be supported by the direction and uncertainty of relevant contrasts, be robust to reasonable modeling choices, and be interpretable in light of the dose levels actually studied.

What the Data Must Support

Let \(D\) denote dose and let \(\mu(D)\) denote the population mean of a continuous response at that dose. For a binary safety outcome, \(\mu(D)\) can instead denote the event probability. A monotone increasing response satisfies \(\mu(D_2)\geq \mu(D_1)\) whenever \(D_2>D_1\); monotone decreasing reverses the inequality. A nonmonotonic claim requires evidence of a change in direction over the range under study, not merely evidence that a flexible curve fits better than a straight line.

In a randomized dose-ranging study, comparisons between dose groups can estimate dose-associated differences, provided outcome definition and assessment are comparable and randomization is maintained. For simple independent groups, the estimated difference between two sample means is \(\bar{x}_b-\bar{x}_a\), and its estimated standard error is \(\sqrt{s_a^2/n_a+s_b^2/n_b}\). This calculation relies on independent observations across groups and a sufficiently accurate sampling approximation; it does not resolve bias from missing outcomes, differential measurement, or post-randomization selection.

A useful first diagnostic is to inspect dose-specific estimates with uncertainty, rather than a fitted curve alone. Ask whether an apparent reversal is large relative to its uncertainty, whether it occurs across more than one adjacent dose contrast, and whether it persists under reasonable analytic choices. A point estimate at the highest dose that is lower than the preceding estimate is a signal to investigate, not a conclusion by itself.

Contrasts as a Direct Check for Curvature

With three equally spaced doses \(D_0,D_1,D_2\), where \(D_1\) is midway between the others, a linear mean-response relationship predicts that the middle mean equals the average of the endpoint means. The contrast \(C=\bar{x}_1-(\bar{x}_0+\bar{x}_2)/2\) measures departure from that prediction. If group observations are independent, the variance of a sample mean is estimated by \(s_i^2/n_i\); therefore, the estimated standard error of this contrast is:

The factors of one quarter arise because the endpoint means have weights of one half, and variances scale by the square of a weight. A positive \(C\) indicates the middle estimate is above the straight-line interpolation; a negative \(C\) indicates it is below. This is a curvature diagnostic, not proof of a particular biological mechanism. To support an inverted-U shape, also check that the middle response exceeds each endpoint, with uncertainty considered for both comparisons.

Separate a Real Pattern from Noise or Model Failure

Sampling variation is a leading explanation for irregular dose means, especially when groups are small or response variability is high. Confidence intervals that span both positive and negative adjacent-dose differences show that the observed direction changes are not precisely determined. Failure to show a reversal is not proof that the true curve is monotone; it may reflect limited information. Conversely, selecting the largest observed mean after examining many doses makes that maximum especially prone to upward sampling error.

Model misspecification is a different problem. A monotone Emax-type model, for example, cannot represent an efficacy downturn if its parameters constrain the response to rise toward a plateau. A poor fit may therefore appear as systematic residuals: observed means consistently above the fitted curve at intermediate doses and below it at high doses, or the reverse. But a flexible polynomial or spline can also track random fluctuations. Compare fitted predictions with dose-specific estimates and uncertainty, inspect residual patterns, and assess whether the inferred direction change survives reasonable alternative models. A single model-selection statistic does not establish the biological credibility of a peak.

Endpoint construction and timing can also create apparent shape changes. A composite outcome may combine components with different dose-response patterns; a response measured at one time may not match a later response. Before interpreting a curve, confirm the endpoint, analysis population, visit, and direction of benefit are the intended ones. Check missingness and influential observations, because a small number of unusual outcomes can materially affect a high-dose mean.

Dose Coverage Limits the Shape You Can Claim

A few tested doses can show that responses differ, yet leave the location and width of a turning region poorly determined. Unequal spacing makes direct comparisons to a straight line especially important: the expected intermediate response under a linear relationship is a dose-weighted interpolation, not necessarily the average of the endpoint means. If \(D_1\) lies between \(D_0\) and \(D_2\), define \(w=(D_1-D_0)/(D_2-D_0)\). The linear interpolation is \((1-w)\bar{x}_0+w\bar{x}_2\). Its contrast with the observed middle mean is \(C=\bar{x}_1-(1-w)\bar{x}_0-w\bar{x}_2\), with standard error \(\sqrt{s_1^2/n_1+(1-w)^2s_0^2/n_0+w^2s_2^2/n_2}\) for independent groups.

Choosing a Diagnostic Approach

The best first approach depends on the question and the dose design. Dose-specific estimates and planned contrasts make few assumptions about curve shape and are transparent, but are less efficient than a correct model and can involve many comparisons. A monotone model can summarize a gradual increase and plateau efficiently, but cannot diagnose a downturn if its form rules one out. Flexible models allow reversals, but their estimates—especially local maxima—can be unstable with few dose levels. A mechanistic PK/PD model can help interpret a plausible exposure-response mechanism, but does not make sparse clinical dose coverage disappear.

Model comparisons should be interpreted alongside uncertainty, residual diagnostics, dose coverage, and scientific plausibility. A sharp estimated peak near the boundary of the tested range deserves particular caution: the true peak may lie outside the range, or the apparent turn may be driven by one uncertain estimate. Sensitivity analysis can show whether the dose ranking changes under plausible choices, but robustness to a set of models does not guarantee that all relevant models or biases have been considered.

Implications for Dose Choice and Common Pitfalls

If a nonmonotonic efficacy pattern is credible, choosing the largest tested dose by default may sacrifice efficacy or increase burden without benefit. The relevant decision should reflect the estimated response and its uncertainty over the supported range, alongside the objectives and constraints already used in dose selection. A safety response that turns in the unfavorable direction can also alter the comparison of doses; diagnosing that shape is a necessary input, not by itself a complete benefit-risk decision.

Avoid claiming an exact optimum from the dose with the largest sample mean. The maximizer of noisy estimates is typically uncertain, and the highest observed point may not equal the maximum of the population curve. Report the tested-dose evidence, uncertainty in contrasts or model predictions, and whether the apparent turning point is interior or at a boundary. Do not extrapolate the curve beyond the observed dose range without explicit justification.

• Mistake: “The high-dose mean is lower, so efficacy falls at high dose.” A lower point estimate alone may be sampling noise. Check the contrast and its uncertainty, and whether the reversal is consistent with other evidence.
• Mistake: “A significant nonlinear model proves the biological mechanism.” Statistical evidence for curvature does not identify its cause. Check endpoint definition, residuals, alternative models, and exposure or biological evidence.
• Mistake: “The fitted peak is the optimal dose.” A curve maximum is not automatically the decision-optimal dose, and sparse coverage may make its location imprecise. Distinguish an estimated peak from a supported dose choice.