Comparing Competing Exposure-Response Models: Worked Analysis

Why Compare Exposure–Response Models?

A relationship between exposure and pharmacodynamic response may rise approximately linearly over the observed range, approach a plateau, or show a gradual transition from baseline to maximum effect. These patterns can be represented by different candidate models, but a visually plausible curve is not enough to choose among them. The comparison must account for fit, model complexity, and predictive performance on subjects not used to fit the model.

This tutorial compares linear, Emax, and sigmoid Emax models on one hypothetical dataset. The goal is not to identify a universally best curve. It is to assess which candidates are supported by the data, whether their predictive performance agrees with their likelihood-based ranking, and how uncertainty about the model affects a prediction.

Candidate Curves and Comparison Assumptions

Let \(C\) denote exposure, measured here in mg/L, and let \(Y\) denote a response measured in percentage points. The three candidate mean functions are:

Here \(E_0\) is the response at zero exposure, \(\beta\) is the linear slope in percentage points per mg/L, \(E_{\max}\) is the maximum increase above baseline in percentage points, \(EC_{50}\) is the exposure producing half that maximum increase, and \(\gamma\) controls the steepness of the sigmoid curve. For the saturating candidates, assume \(EC_{50}>0\), \(E_{\max}>0\), and, for sigmoid Emax, \(\gamma>0\). These equations describe an increasing response; a decreasing response requires an appropriate sign convention or a different parameterization.

For the worked comparison, assume each of 30 independent hypothetical subjects contributes one response and one exposure summary. Conditional on exposure, responses have a normal distribution with the candidate mean function and a common residual variance within each fitted model. Exposure is treated as measured without error. The candidates use the same subjects, response scale, and likelihood family; each likelihood is maximized over its model parameters, including residual variance. These assumptions make the likelihood criteria comparable. They would need reconsideration for repeated observations, correlated responses, influential outliers, or substantial exposure measurement error.

For a model with maximized log likelihood \(\ell\), \(K\) estimated parameters, and \(n\) independent observations, the criteria used below are:

Smaller AICc or BIC indicates a better trade-off between fit and complexity under that criterion; neither is a probability that a model is true. AICc requires \(n>K+1\). The parameter count includes the residual variance, and all models must use the same observations and likelihood definition for the comparison to be meaningful.

From AICc Differences to Relative Weights

Akaike weights provide a normalized summary of relative support among the candidate models. If \(\Delta_m=AICc_m-\min_j AICc_j\), define the unnormalized relative support as \(\exp(-\Delta_m/2)\), then divide by the sum across candidates. The resulting weights are nonnegative and sum to one.

This is a useful weighting scheme for combining predictions, but the weights are not automatically posterior model probabilities. Their interpretation depends on the candidate set and the information criterion, and should not be extended to models that were selected through an unaccounted-for search.

Check Predictive Performance by Subject

Information criteria summarize fit and penalize complexity, but prediction is a distinct target. In subject-level cross-validation, all data from a held-out subject are excluded from fitting, the model is fitted using the remaining subjects, and a predictive score is calculated for the held-out subject. If subjects contribute repeated records, those records must stay together in the same fold; splitting one subject’s records between training and validation can leak subject-specific information.

For a normal predictive distribution with mean \(\hat{\mu}_{i,-i}\) and standard deviation \(\hat{\sigma}_{i,-i}\), the held-out log predictive density for response \(y_i\) is \(\log f(y_i\mid\hat{\mu}_{i,-i},\hat{\sigma}_{i,-i})= -\frac12\log(2\pi)-\log(\hat{\sigma}_{i,-i})-\frac12[(y_i-\hat{\mu}_{i,-i})/\hat{\sigma}_{i,-i}]^2\). A higher average held-out log predictive density is better. Cross-validated RMSE is another useful summary, with lower values better, but it assesses errors on the response scale rather than the full predictive density.

Model-Averaged Predictions and Their Uncertainty

Predictions can be combined on the response scale even when model parameters have different meanings. For instance, \(\beta\) is a linear slope, whereas \(EC_{50}\) is a half-maximum exposure; averaging these parameters across models would not produce a coherent quantity. Instead, calculate each model’s predictive mean at the exposure of interest and combine those means using the model weights.

If model \(m\) assigns weight \(w_m\), predictive mean \(\mu_m\), and predictive variance \(V_m\), the model-averaged mean is \(\mu_{\mathrm{avg}}=\sum_m w_m\mu_m\). Its predictive variance includes uncertainty within models and variation between their means:

The first term is the weighted within-model predictive variance; it can include residual variability and parameter uncertainty, depending on how each model’s predictive distribution was constructed. The second is the between-model contribution. Both terms have squared response units.

Diagnostics, Alternatives, and Limitations

A model comparison is only as credible as its fits and validation design. Before interpreting a ranking, check convergence, parameter plausibility, and residual behavior using the relevant PD diagnostics. For nonlinear candidates, inspect whether estimates are at constraints or poorly identified: a very large \(EC_{50}\), for example, may indicate that the observed exposure range does not reveal the plateau. Plot fitted curves over the observed range and examine whether one or two influential subjects drive the ranking. A candidate with a favorable criterion but systematic residual structure may be inadequate rather than a useful winner.

AICc, BIC, and cross-validation answer related but not identical questions. AICc targets relative expected predictive information loss with a finite-sample correction; BIC applies a stronger, sample-size-dependent complexity penalty and is often used for a different asymptotic model-selection objective. Subject-level cross-validation estimates performance under its particular holdout design and can reveal poor out-of-sample behavior. When rankings disagree, examine the prediction target, fold variability, candidate complexity, and assumptions rather than selecting whichever number is most convenient.

A single best-model prediction is simple, but it ignores uncertainty from plausible alternatives. Weighted model averaging represents that uncertainty if the candidate set is scientifically defensible and the weights are appropriate to the goal. It cannot repair a set that excludes the relevant response shape, and it should not be used to justify extrapolation far beyond observed exposures. At such exposures, candidate curves can diverge sharply despite similar in-range fit. The introductory material on predictive stacking discusses an alternative way to construct predictive combination weights; the present calculation uses relative AICc weights to demonstrate uncertainty decomposition.

Common Pitfalls

• Comparing incompatible likelihoods. AICc differences are not meaningful if candidates use different observations, response definitions, or likelihood normalizations.
• Splitting repeated observations across folds. This lets training data contain information from a validation subject. Keep all of each subject’s records in one fold.
• Treating weights as truth probabilities. AICc weights summarize relative support within the supplied candidate set; they do not prove that one model is true.
• Reporting only within-model uncertainty. Omitting the between-model term understates predictive variance when plausible models give different means.
• Trusting a ranking despite failed diagnostics. Convergence problems, boundary estimates, influential subjects, or systematic residual patterns can invalidate a nominally favorable criterion.