Reporting Sensitivity Analyses and Their Impact on Conclusions

Why Report Sensitivity Analyses?

A primary pharmacometric analysis is conditional on choices about the model and data. A sensitivity analysis asks whether a conclusion changes under a different, defensible choice—for example, a different residual-error assumption or a plausible rule for handling a data feature. The purpose is not to make every alternative agree with the primary analysis. It is to show which findings persist, which change, and what those changes mean for interpretation.

A report should connect three things: the assumption varied, the resulting change in estimates or uncertainty, and the consequence for the stated scientific or decision question. A list of alternative fits without this connection can be technically detailed yet uninformative. Conversely, saying that results were “robust” without showing what was compared gives readers no basis to assess that claim.

Definition: A sensitivity analysis evaluates how results or conclusions change when one or more plausible analysis assumptions or decisions are altered. The primary analysis is the analysis designated as the main basis for the reported result. A conclusion is robust to a specified alternative when its substantive interpretation and decision consequence remain materially unchanged under that alternative. Robustness is always relative to the alternatives examined; it does not establish that the model is correct.

For a meaningful comparison, first establish what is being held fixed. Whenever possible, the primary and alternative analyses should address the same population, outcome, estimand, and parameter scale. If these differ, a numerical contrast may mix the effect of the changed assumption with a change in the question being answered. State any such differences explicitly rather than presenting the estimates as directly comparable.

The alternative should also be plausible and targeted. It should examine an assumption that could reasonably affect the inference, not an arbitrary set of changes chosen because they produce a preferred result. Prespecifying sensitivity analyses and their decision rules helps distinguish planned robustness checks from exploratory analyses. If an alternative is added after seeing the primary result, report that timing accurately.

Quantifying Changes Without Losing the Context

For a parameter \(\theta\) reported on the same scale in both analyses, the signed absolute change is \(\hat{\theta}_{A}-\hat{\theta}_{P}\), where \(P\) denotes the primary analysis and \(A\) the alternative. When the primary estimate is not near zero, the relative change can also be useful:

\text{Relative change (\%)}= \frac{\hat{\theta}_{A}-\hat{\theta}_{P}}{|\hat{\theta}_{P}|}\times 100\%.

The denominator uses the magnitude of the primary estimate so that the signed numerator indicates the direction of change. A relative change is unstable or unhelpful when the primary estimate is near zero; report the absolute change and its units instead. For a parameter estimated on a transformed scale, state whether the contrast is on that scale or after back-transformation. For instance, a difference in log-clearance is not itself a difference in clearance units.

Point estimates alone do not establish that a conclusion is robust. Report uncertainty using a consistent method where possible, and identify the decision rule that gives the result meaning. Do not infer that two estimates differ simply because their confidence intervals do not overlap, or that they are equivalent because the intervals overlap. Interval overlap is not a formal test of the difference, and neither finding substitutes for a prespecified criterion for material change.

Reporting principle: For each relevant alternative, report the changed assumption, estimates and uncertainty on an interpretable scale, the comparison with the primary result, and whether the same scientific or decision conclusion follows. Define “material” in relation to the parameter’s units and the decision context, not by a universal percentage threshold.

Worked Example: Comparing Clearance Estimates

In a hypothetical population PK analysis, the primary model estimates typical clearance as 5.20 L/h, with a 95% confidence interval of 4.60 to 5.90 L/h. A prespecified alternative residual-error assumption gives 5.55 L/h, with a 95% confidence interval of 4.85 to 6.35 L/h. Both fits use the same population and parameterization, and both are considered adequate for this comparison. Calculate and interpret the estimate change.

The signed absolute change is \(5.55-5.20=0.350\) L/h. The relative change is \(0.350/|5.20|\times100\%=6.73\%\). Thus, under the alternative assumption, estimated typical clearance is higher by 0.350 L/h, or 6.73% relative to the primary estimate.

The intervals describe uncertainty for each estimate; they do not show that the estimates are identical. The report should say that the point estimate changes modestly on the clearance scale, provide both intervals, and explain whether a 0.350 L/h shift would affect the intended interpretation or decision. Without a context-specific threshold or downstream consequence, it would be unjustified to label that shift clinically negligible solely because it is 6.73%.

From Estimate Changes to Decision Consequences

A sensitivity analysis is most useful when it evaluates the conclusion the analysis was intended to support. That conclusion might concern the direction or magnitude of a covariate effect, whether an interval excludes a null value, or whether a model-based quantity crosses a prespecified decision threshold. The report should name the criterion before describing whether it was met. A change in a p-value or interval position is not, by itself, a complete account of clinical importance.

The following example uses a normal-approximation confidence interval. Its assumptions are that the estimator is approximately normally distributed, the reported standard error adequately estimates its sampling variability, and the same interval procedure is used for both analyses. The example is illustrative; other inferential procedures may be appropriate for a particular model.

Worked Example: A Covariate Finding That Depends on an Assumption

Suppose a hypothetical analysis estimates the change in a biomarker per 10-unit increase in a covariate. The units of the effect are biomarker units per 10 covariate units. The primary estimate is \(0.42\), with \(SE=0.16\); an alternative, defensible data-handling assumption gives an estimate of \(0.29\), with \(SE=0.17\). Use 1.96 as the normal critical value for approximate 95% intervals. The prespecified inferential criterion is that the interval exclude zero.

For the primary analysis, the interval is \(0.42\pm1.96(0.16)=0.42\pm0.3136\), or approximately \([0.106,0.734]\). It excludes zero, so the primary analysis meets the stated criterion for evidence of a positive covariate association.

For the alternative, the interval is \(0.29\pm1.96(0.17)=0.29\pm0.3332\), or approximately \([-0.043,0.623]\). It includes zero, so the alternative does not meet that criterion. The point estimate decreases by \(0.29-0.42=-0.13\) biomarker units per 10 covariate units; the relative change is \(-0.13/0.42\times100\%=-31.0\%\).

The appropriate conclusion is that the estimated association remains positive as a point estimate, but the inferential conclusion under the stated interval-exclusion rule depends on the data-handling assumption. This does not prove there is no association under the alternative, nor does it prove the assumptions cause the change. It shows that the evidence is not stable to this specified alternative.

This comparison is conditional on the interval approximation and on the two analyses targeting the same effect. If the alternative changes the estimand or excludes a substantively different population, the conclusion should be framed as a comparison of different targets, not simply as a sensitivity of one estimate. Likewise, if the estimates use different uncertainty methods, disclose that fact and avoid attributing interval changes solely to the varied assumption.

A Short Justification for Comparing Estimates

The relative-change calculation is a rescaling of the absolute difference, not a new measure of statistical evidence. It expresses the difference as a fraction of the primary estimate’s magnitude. Starting with the absolute difference \(d=\hat{\theta}_{A}-\hat{\theta}_{P}\), divide by \(|\hat{\theta}_{P}|\) to measure that difference relative to the primary magnitude, then multiply by 100 to express it as a percentage:

d=\hat{\theta}_{A}-\hat{\theta}_{P}, \qquad 100\frac{d}{|\hat{\theta}_{P}|} = \frac{\hat{\theta}_{A}-\hat{\theta}_{P}}{|\hat{\theta}_{P}|}\times100\%.

This quantity retains the direction of change through the sign of \(d\), but it has no stable interpretation if \(|\hat{\theta}_{P}|\) is close to zero. For example, a change from 0.02 to 0.04 is a 100% relative increase, although the absolute change is only 0.02 in the parameter’s units. Therefore, report the absolute difference alongside the percentage, and prioritize the absolute scale when the primary value is small or when the parameter has a natural clinical unit.

A conclusion can also be evaluated against a practical threshold rather than a null value. In that case, state the threshold, how it was chosen, and whether the relevant estimate or uncertainty measure crosses it under each analysis. Do not substitute a threshold discovered after seeing the results for a prespecified decision rule without identifying it as exploratory.

Worked Example: A Prediction That Crosses a Decision Threshold

A hypothetical dosing assessment asks whether at least 70% of patients are predicted to attain a target exposure. Under the primary model, the predicted proportion is 74%; under a plausible alternative model structure, it is 68%. Both predictions refer to the same target, population, and dosing regimen. For this illustration, treat 70% as the prespecified decision threshold; no uncertainty interval is supplied.

The absolute change is \(68\%-74\%=-6\) percentage points. Relative to the primary prediction, the change is \((-6/74)\times100\%=-8.11\%\). The primary prediction exceeds the threshold by 4 percentage points, whereas the alternative is 2 percentage points below it.

Thus, the decision based on the stated point-prediction rule changes: the primary analysis meets the target criterion and the alternative does not. The result is assumption-dependent for this decision. Because uncertainty intervals are not provided, the report must not imply that either predicted proportion is known precisely or that the threshold has been conclusively met or missed.

Diagnostics, Alternatives, and Limits

Before calling a conclusion robust, verify that the comparison itself is interpretable. Useful checks include:

Confirm that the alternative was plausible, clearly described, and targeted to an identifiable assumption.
Check that population, outcome, estimand, parameterization, units, and uncertainty procedure are comparable, or explain how they differ.
Verify that both analyses produced estimates suitable for the comparison. A failed or materially incomplete fit cannot be treated as evidence of stability.
Compare the quantities that drive the scientific claim or decision, not only a convenient parameter or a global fit summary.
State whether the decision rule was prespecified and distinguish a change in statistical evidence from a change in clinical interpretation.

Sensitivity analysis differs from model comparison and selection: its central question is how conclusions respond to plausible assumptions, not which candidate should be selected as the preferred model. The selection rationale explains why a model is used as primary; sensitivity reporting then shows what happens when a relevant choice is varied. A full alternative-analysis report is also different from a single robustness label, because readers need to see the direction and magnitude of change.

There are limits to what such analyses can establish. Examining a few alternatives cannot cover every possible source of misspecification. Agreement may arise because the alternatives share the same problematic assumption, and disagreement does not identify which analysis is closer to the truth. Estimates can also vary because of sampling uncertainty, not solely because an assumption changed. Report the scope of the alternatives and avoid general claims such as “the results are robust” unless the statement specifies which findings were stable and under what changes.

Common Reporting Mistakes

Calling analyses robust because point estimates look similar. A small numerical change may cross a decision threshold. Report the criterion and consequence, not just visual similarity.
Using percentage change near zero. The denominator can make the percentage enormous or undefined. Give the absolute change in units and explain why a relative change is not informative.
Treating overlapping confidence intervals as proof of no difference. Overlap does not establish equivalence. Compare the estimates and their uncertainty in context, using an appropriate procedure if a formal contrast is required.
Reporting only a changed p-value. A threshold crossing can depend on small changes in uncertainty. Include effect estimates, intervals, units, and the decision rule.
Describing a changed conclusion as proof that one assumption is correct. Sensitivity analysis reveals dependence on the examined choices; it does not identify the true model.

Key takeaway: A useful sensitivity report makes the path from assumption to estimate to decision visible. Quantify changes on an interpretable scale, show uncertainty, and state precisely whether the conclusion is stable or assumption-dependent.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A primary estimate is 2.00 mg/L and an alternative estimate is 1.80 mg/L. Calculate the signed absolute and relative changes.
Why is a relative change potentially misleading when the primary estimate is close to zero?
A primary 95% interval excludes zero, while an alternative interval includes zero. What is a defensible conclusion if the prespecified rule requires exclusion of zero?
What should be checked before directly comparing estimates from two analyses?

Solutions. (1) The absolute change is \(1.80-2.00=-0.20\) mg/L. The relative change is \((-0.20/2.00)\times100\%=-10.0\%\): the alternative estimate is 10.0% lower than the primary estimate. (2) The primary estimate is the denominator, so a small denominator can produce a very large percentage from a small absolute difference; if it is zero, the relative change is undefined. Report the absolute change with units. (3) The primary analysis meets the stated criterion and the alternative does not. The inferential conclusion is sensitive to the specified alternative; this does not prove the effect is absent under the alternative. (4) Check whether the analyses target the same population, outcome, estimand, parameter scale, and units, and whether their uncertainty methods are comparable. Disclose differences that prevent a like-for-like interpretation.