Why Test a Claim Against Model Structure?
A label statement supported by a model is conditional on how the model represents the data. For example, a predicted exposure ratio may depend on whether renal function is represented by categories or a continuous function, whether age modifies the renal-function effect, and whether a small number of observations strongly affect the fit. A single fitted model cannot show how consequential those choices are.
The purpose of a sensitivity analysis is not to search among models until one produces the preferred wording. It is to determine whether the same scientifically relevant claim remains supported across a set of plausible, defensible alternatives. The claim, target population, prediction time, dose regimen, and decision threshold must be held consistent where possible; otherwise, an apparent difference may reflect a changed question rather than changed model structure.
A claim should be made operational before testing it. In the examples below, the hypothetical claim is that exposure at a specified dose in a target population is no more than 1.25 times exposure in a reference population. The threshold of 1.25 is illustrative, not a universal clinical standard. A real threshold must be justified for the drug, clinical context, and intended statement. Earlier material on communicating model uncertainty distinguishes the prediction itself from uncertainty around that prediction; this tutorial focuses on how both may change when model assumptions are varied.
Specify What Must Remain Comparable
Let \(\delta\) be the model-estimated contrast in mean log exposure between a target and a reference population under a specified regimen. The corresponding geometric-mean exposure ratio is \(R=\exp(\delta)\). A threshold claim \(R\leq 1.25\) is equivalent to \(\delta\leq\log(1.25)\), which is approximately \(0.2231\). This equivalence lets analysts compare models on either scale, provided they use the same estimand.
For a log-scale estimate \(\hat{\delta}\) with standard error \(SE\), an approximate two-sided 90% confidence interval, under a large-sample normal approximation, is \(\hat{\delta}\pm1.645SE\). Exponentiating both limits gives an interval for the ratio. This is valid because the exponential function is strictly increasing. The interval describes uncertainty under the fitted model and its assumptions; it does not include uncertainty over which model structure is correct.
The normal approximation assumes that the estimator is approximately normally distributed on the log scale and that its standard error is adequately estimated. In small or highly unbalanced data sets, or with a complicated nonlinear model, that approximation may be poor; simulation or another suitable inferential method may be needed. In every case, the estimand must also be clear: a conditional contrast at a particular covariate value is not automatically interchangeable with a population-standardized contrast.
Comparing Plausible Structural Models
Alternative structures should arise from scientific knowledge, study design, or visible limitations of the primary model. For a continuous predictor such as kidney function, a linear effect on log exposure might be compared with categories or a smooth nonlinear function if there is a plausible reason to question linearity. Models should use the same endpoint, target population, regimen, and contrast wherever possible. Their predictions and uncertainty can then be compared directly.
Worked Example: A Threshold Claim Under Alternative Structures
Hypothetical data and setup. A pharmacometric analysis predicts the ratio of exposure at a specified dose in a target renal-function population to exposure in a reference population. Three defensible specifications are considered: a linear renal-function effect, categories of renal function, and a smooth nonlinear effect. Suppose the fitted log-scale contrasts and standard errors are as follows.
| Specification | \(\hat{\delta}\) | \(SE\) |
|---|---|---|
| Linear renal-function effect | 0.148 | 0.055 |
| Categorical renal-function effect | 0.190 | 0.070 |
| Smooth nonlinear effect | 0.235 | 0.085 |
Model assumptions. Each model is assumed to describe the relevant exposure data adequately, with the specified renal-function relationship and appropriate residual and random-effects distributions. The target and reference predictions are defined for the same population and regimen. The intervals below use the approximate normal method and a 90% confidence level; these assumptions must be checked before using the calculations.
Linear specification. Its point estimate is \(\exp(0.148)\approx1.160\). The log-scale interval is \(0.148\pm1.645(0.055)=0.148\pm0.0905\), or approximately \([0.0575,0.2385]\). Exponentiating gives \([\exp(0.0575),\exp(0.2385)]\approx[1.059,1.269]\). The point estimate is below 1.25, but the interval extends above 1.25.
Categorical specification. The predicted ratio is \(\exp(0.190)\approx1.209\). The log interval is \(0.190\pm1.645(0.070)=0.190\pm0.1152\), or approximately \([0.0749,0.3051]\). The ratio interval is approximately \([1.078,1.357]\), which also extends above the threshold.
Smooth specification. The predicted ratio is \(\exp(0.235)\approx1.265\), above the threshold. Its log interval is \(0.235\pm1.645(0.085)=0.235\pm0.1398\), or approximately \([0.0952,0.3748]\). The ratio interval is approximately \([1.100,1.455]\).
The point predictions range from about 1.160 to 1.265, and the smooth model’s point prediction exceeds 1.25. Thus the claim is not robust as a statement that the predicted ratio is below 1.25 under every plausible structure. The two intervals that cross the threshold also show that, within those individual models, uncertainty is not confined to values below it. This result does not prove that the smooth model is correct or that the true ratio exceeds 1.25; it identifies structural sensitivity that should qualify the conclusion.
A short justification explains why the threshold comparison is equivalent on the two scales. Since the derivative of \(\exp(\delta)\) is \(\exp(\delta)>0\), exponentiation preserves order. Therefore \(\hat{\delta}<\log(1.25)\) exactly when \(\exp(\hat{\delta})<1.25\). The same order-preserving transformation applies to interval limits, but it does not eliminate uncertainty or model dependence.
Covariate Specifications and the Target Population
Covariate sensitivity is interpretable only after defining the prediction being compared. In a model with an interaction, the exposure contrast may vary with the covariate value. In a model without an interaction, an adjusted contrast may be constant on the modeled scale, but the population-level prediction can still depend on how covariates are distributed in the target population. A covariate-adjusted coefficient and a standardized population prediction are not necessarily the same quantity.
Worked Example: Age Adjustment and an Interaction
Hypothetical model and data. Consider a model for log exposure with renal-function indicator \(R\), where \(R=1\) denotes the target group and \(R=0\) the reference group, and centered age \(a=(\text{age}-50)/10\). Suppose a fitted model is \(\eta=\beta_0+\beta_RR+\beta_Aa+\beta_{RA}Ra\), with estimates \(\hat{\beta}_R=0.100\), \(\hat{\beta}_A=0.030\), and \(\hat{\beta}_{RA}=0.020\). These are hypothetical estimates; the contrast is evaluated at age 70, so \(a=2\).
Assumptions. The log-exposure mean is adequately represented by this linear predictor over the age range considered, and the interaction is included because it is a plausible alternative to a constant renal-function contrast. Both groups must have adequate age support near age 70; otherwise this prediction would rely substantially on extrapolation. The other model terms cancel in the within-age contrast.
At age 70, the target-minus-reference log contrast is \(\hat{\delta}(70)=\hat{\beta}_R+\hat{\beta}_{RA}a=0.100+0.020(2)=0.140\). The predicted ratio is \(\exp(0.140)\approx1.150\). If the interaction is omitted and the fitted renal-function coefficient is hypothetically \(0.120\), the ratio at the same age under that alternative model is \(\exp(0.120)\approx1.128\). The estimates are close in this example, but their difference should still be considered alongside their uncertainty and the fit of each model.
Suppose, however, that the target group is much older than the reference group. A crude comparison of their average exposures could combine the renal-function contrast with an age effect. It would then answer a different question from the age-specific contrast just calculated. For a population claim, predictions should be standardized over the intended target population’s age distribution, using each model’s prediction function and the same target distribution. That keeps the estimand aligned while testing the covariate specification.
Useful checks include plotting the covariate distributions by group, confirming overlap in the region used for prediction, and examining whether the modeled relationship is supported by data rather than extrapolation. A change in prediction after adding a covariate is not automatically evidence that the simpler model was biased: it may also reflect a changed estimand, limited overlap, or uncertainty in the fitted coefficients. State which explanation is plausible and quantify the impact on the claim.
Influential Observations: Diagnose Before Excluding
An influential observation is one whose presence can materially change a fitted result. Influence is not the same as being erroneous. A measurement can be unusual and valid, while a data error may be influential or not. An exclusion requires a defensible reason, such as a documented protocol deviation or verified measurement failure—not merely that removing the observation makes the claim more favorable.
Worked Example: Leave-One-Out Sensitivity
Hypothetical observations and setup. Suppose five independent subjects each contribute a valid paired log-exposure contrast, \(d_i\), between the target and reference conditions. The values are \(0.08,\ 0.10,\ 0.12,\ 0.14,\) and \(0.40\). Assume the subject-level contrasts are measured comparably and that the simple mean is the prespecified estimator of the average log contrast.
Assumptions. Subjects are independent, the contrasts represent the same target comparison, and there is no documented data-quality reason to exclude any subject. The mean is appropriate for the intended estimand. With only five subjects, this calculation is illustrative and does not establish a reliable population estimate.
The full-sample mean log contrast is \(\bar d=(0.08+0.10+0.12+0.14+0.40)/5=0.84/5=0.168\). Its ratio-scale estimate is \(\exp(0.168)\approx1.183\). The largest contrast, \(0.40\), is a candidate for influence checking. Removing it only as a diagnostic gives \(\bar d_{(-5)}=(0.08+0.10+0.12+0.14)/4=0.44/4=0.110\), with ratio \(\exp(0.110)\approx1.116\).
The leave-one-out change is \(0.168-0.110=0.058\) on the log scale, and the corresponding point prediction changes from about 1.183 to 1.116. That is a consequential sensitivity. It does not justify reporting the reduced estimate as primary: absent a valid exclusion reason, the full data remain relevant. The appropriate conclusion is that this small data set produces a claim that is sensitive to one subject and merits careful review and appropriately qualified interpretation.
In larger fitted models, influence can be assessed with case-deletion changes in the estimand, parameter-specific influence measures, and residual diagnostics appropriate to the model. Review data provenance, dosing and sampling records, and model diagnostics for the influential cases. Robust estimation may be a useful alternative when its target and assumptions are appropriate, but it is not a license to conceal sensitivity: report how its estimate relates to the prespecified primary analysis.
How to Conduct and Report the Assessment
Specify the estimand, target population, reference, regimen, time point, scale, and any threshold. State whether the claim concerns a point prediction or uncertainty as well.
Vary structural features with scientific or data-based justification, such as functional form or a plausible covariate interaction. Do not choose alternatives solely because of their result.
Assess fit, residual behavior, data support, covariate overlap, and influential observations. Confirm that each model is predicting the same estimand for the same target population.
Present the prediction and an appropriate uncertainty interval for each model on an interpretable scale. Separate within-model uncertainty from variation across structural choices.
Identify whether the threshold conclusion is stable, conditional on a particular assumption, or not supported across the plausible alternatives. Explain any consequential influence or extrapolation.
A useful summary table reports each alternative, its rationale, its predicted contrast, its uncertainty, and whether it changes the claim. Diagnostics are not a pass-or-fail ritual: residual patterns, poor overlap, unstable estimates, or a large case-deletion change are signals to investigate and disclose. If an alternative model fits poorly or conflicts with established mechanism, it need not receive equal scientific weight, but the reason for discounting it should be explicit.
Alternatives, Limitations, and Common Pitfalls
A single primary model is efficient and may be well justified, but by itself it does not measure structural robustness. A sensitivity set of prespecified alternatives makes conditionality visible, although it cannot guarantee that every plausible structure has been considered. Model averaging is another possible approach when models and weights have a defensible basis; it summarizes across selected models but does not remove the need to define the estimand or justify the candidate set. These approaches answer related but not identical questions.
Sensitivity analyses are limited by the data and by the alternatives considered. Several models can agree because they share the same misspecification. A wide range of estimates may reflect weak information, poor covariate overlap, or genuine structural dependence. Do not present the range of point estimates as a confidence interval: it is a summary of selected model results, not a probability statement. Conversely, a narrow range does not prove the claim is true.
- Pitfall: choosing the favorable model. Wrong reasoning: “The categorical model is below the threshold, so it supports the claim.” Correction: justify the model independently of its result and show how other plausible structures affect the same estimand.
- Pitfall: treating an interval crossing a threshold as proof of failure. Wrong reasoning: “Some values exceed 1.25, so the true ratio exceeds 1.25.” Correction: an interval crossing the threshold indicates uncertainty under that model; it does not establish which values are true.
- Pitfall: deleting an influential case because it changes the conclusion. Wrong reasoning: “The claim is clearer without this subject.” Correction: retain valid observations in the primary analysis; exclude only for a documented, defensible reason and report the sensitivity.
- Pitfall: comparing unlike predictions. Wrong reasoning: “The coefficients differ, therefore the claim is structurally unstable.” Correction: first verify that the models target the same population, covariate values or standardization, dose, endpoint, and contrast.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
- A model estimates a log exposure contrast of \(0.20\). Is its ratio-scale estimate below 1.25, and what calculation supports your answer?
- Two models give different renal-function coefficients. Name two checks needed before interpreting this as a change in the label claim.
- A 90% confidence interval crosses a prespecified exposure-ratio threshold. What does that establish, and what does it not establish?
- An observation strongly changes the predicted ratio when removed, but the record shows no protocol or measurement error. Should it be excluded from the primary analysis?
- Why is the range of predictions from several sensitivity models not itself a confidence interval?
Solutions. (1) \(\exp(0.20)\approx1.221\), which is below 1.25 as a point estimate; it does not alone establish that uncertainty is below the threshold. (2) Check that both models use the same estimand and target population, and assess covariate overlap and model fit. (3) It establishes that the interval under that model includes values on both sides of the threshold; it does not prove the true ratio exceeds the threshold or that the model is correct. (4) No. Retain the valid observation in the primary analysis and report the influence sensitivity. (5) The range reflects the selected models’ point predictions, not repeated-sampling uncertainty under a defined inferential procedure.