Separating Statistical Significance From Clinical Dose Relevance

Why Statistical Significance Is Not a Dose Justification

A dose-response analysis can estimate a contrast precisely enough to show that it differs from zero, while the estimated difference is too small to matter clinically. Conversely, a potentially important contrast can be estimated imprecisely, leaving uncertainty about whether it is real and whether it reaches a meaningful size. These are different questions, and a dose decision should not treat a small p-value as an answer to both.

The relevant comparison is between a dose contrast and a clinically important threshold defined on the endpoint scale. That threshold must be chosen before interpreting the results, using clinical and development context. It is not a universal constant, and it does not replace examination of uncertainty, model adequacy, or other evidence relevant to dose selection.

If the original endpoint is one where lower values are better, such as symptom burden, orient the contrast in the beneficial direction before applying this convention. State the endpoint, units, dose comparison, direction of benefit, and threshold together. A threshold of 3 points has no interpretable meaning without knowing which scale those points are on and what difference they represent.

Two Boundaries, Two Questions

For a two-sided test of no dose contrast, the hypotheses are \(H_0:\Delta_{H:L}=0\) and \(H_a:\Delta_{H:L}\ne 0\). At significance level \(\alpha\), a small p-value is evidence against the zero-contrast null under the model and test assumptions. It does not establish that the contrast exceeds \(\delta\). A clinically important contrast, by contrast, lies at or above the threshold: \(\Delta_{H:L}\geq\delta\).

A confidence interval makes the distinction visible. If a 95% confidence interval lies entirely above zero but entirely below \(\delta\), the estimated contrast is statistically distinguishable from zero yet is not supported as reaching the prespecified threshold. If the interval lies entirely above \(\delta\), the data support a contrast exceeding the threshold at the corresponding confidence level. If the interval crosses \(\delta\), the estimate alone may be above the threshold, but the uncertainty does not resolve whether the contrast exceeds it. A non-significant result is not proof that the contrast is zero or clinically unimportant.

These interpretations assume the contrast, threshold, direction, and analysis have been specified appropriately; the dose-response model is sufficiently adequate for the comparison; and the reported standard error and interval reflect the relevant uncertainty. Confidence intervals describe the long-run coverage of the procedure under its assumptions, not a probability that the fixed true contrast lies in this particular interval.

Uncertainty in a Model-Predicted Dose Contrast

When dose means come from a fitted dose-response model, their uncertainty is generally correlated. The contrast standard error must account for that covariance; subtracting two standard errors or treating predicted means as independent is not generally correct. For model parameters \(\theta\), an approximate variance for a smooth contrast follows from the delta method.

Here \(g(\theta)\) is the model-implied contrast, \(\hat{\theta}\) is the estimated parameter vector, and \(\widehat{\mathrm{Cov}}(\hat{\theta})\) is its estimated covariance matrix. The approximation uses a local linearization of \(g\) near \(\hat{\theta}\); it is most reliable when the estimator is approximately normal and the contrast is not so nonlinear near the estimate that this linearization is poor. A typical approximate 95% interval is \(\widehat{\Delta}\pm1.96SE(\widehat{\Delta})\).

For two model-predicted means, \(\hat{\mu}_H\) and \(\hat{\mu}_L\), the same point is expressed as \(\widehat{\mathrm{Var}}(\hat{\mu}_H-\hat{\mu}_L)= \widehat{\mathrm{Var}}(\hat{\mu}_H)+\widehat{\mathrm{Var}}(\hat{\mu}_L) -2\widehat{\mathrm{Cov}}(\hat{\mu}_H,\hat{\mu}_L)\). The covariance matters because both predictions may depend on the same fitted parameters. The following examples use hypothetical estimates and approximate normal intervals; the stated calculations assume the model-based standard errors are appropriate for the specified contrasts.

Using the Results Responsibly

The confidence interval relative to both zero and \(\delta\) is often more informative than reporting a p-value alone. A useful interpretation names the contrast and its units, reports the estimate and interval, states the prespecified threshold, and distinguishes the two questions. For example: “The higher dose was estimated to improve the score by 1.20 points (95% CI 0.906 to 1.494); this was distinguishable from zero, but the interval remained below the prespecified 3.0-point clinical threshold.”

A formal test can also target the clinical threshold rather than zero. For example, evidence that \(\Delta>\delta\) can be assessed with a one-sided test of \(H_0:\Delta\leq\delta\) against \(H_a:\Delta>\delta\), at a prespecified \(\alpha\). This asks a different question from the conventional test of \(H_0:\Delta=0\). A lower one-sided confidence bound above \(\delta\) corresponds to evidence against that threshold null at the matching level, subject to the same assumptions. The threshold test should not be selected after looking at the observed estimate.

Model-based intervals require particular care when predictions are extrapolated beyond observed doses, the dose-response shape is poorly supported, or parameter estimates are highly correlated or weakly identified. Inspect fit and uncertainty in the region of the dose contrast, and check whether plausible alternative model specifications materially change the estimate or interval. If several dose contrasts or endpoints are examined, multiplicity and the selection process can make nominal intervals or p-values less reliable as summaries of the evidence. Also report the scale: a threshold defined for one endpoint, population, or treatment duration cannot automatically be carried over to another.

The main alternative to a threshold-based interval assessment is to report statistical significance alone. That is simpler but does not answer whether the effect size matters. Another approach is to report only whether the point estimate exceeds \(\delta\); this ignores uncertainty and can make a noisy estimate look decisive. Reporting the estimate and interval against both boundaries preserves more information. None of these summaries repairs a poorly chosen threshold, a misspecified model, or a contrast that is not relevant to the clinical question.

Common Mistake and Key Takeaway

Wrong reasoning: “The higher dose has \(p<0.05\), so its benefit is clinically important and the dose is justified.” A p-value for a zero-contrast test does not compare the estimated effect with a clinical threshold. Correction: report the contrast and uncertainty in endpoint units, compare the interval with the prespecified threshold, and limit the conclusion to what that comparison supports.