What a Sensitivity Analysis Adds
A primary population PK analysis provides the main estimates and conclusions for the stated regulatory question. A sensitivity analysis examines whether those conclusions depend on a plausible alternative choice, such as a different analysis dataset or a different modeling assumption. The purpose is not to generate a collection of competing “best” answers. It is to show which conclusions remain stable, which change, and why that change matters.
A robustness claim is therefore bounded: it refers to the particular alternatives examined and the particular conclusions compared. Similar estimates across a few alternatives do not establish that the model is correct under every possible assumption. Conversely, a difference does not automatically invalidate the primary analysis; it calls for an explanation of what changed, whether the alternative is credible, and whether a decision would be affected.
The primary analysis should be clearly identified, including its analysis set and main assumptions. For each sensitivity analysis, report what differed and what remained the same. A comparison is interpretable only when readers can tell whether changes in the result follow from the intended alternative or from several unreported changes made together.
Plan the Comparison Around the Decision
Before comparing estimates, state the decision or conclusion the analysis is meant to inform. Examples include whether a covariate effect is large enough to affect dose selection, whether a parameter estimate remains within a clinically acceptable range, or whether the overall interpretation changes when a subset of records is excluded. The decision context determines which outputs deserve emphasis. There is no universal percentage difference that defines a result as robust.
For each alternative, identify the altered feature precisely. “Alternative dataset” is not enough: say which records were added, excluded, or handled differently, and give the resulting sample size when relevant. “Alternative model” is similarly vague: name the assumption or model feature changed. Also state whether the alternative was planned or arose during analysis, and give a concise rationale for examining it.
Name the primary analysis, its data cutoff or analysis set, and the key estimates or conclusions used as the comparison point.
Specify exactly what assumption or dataset changed, why the alternative is plausible, and what was held constant.
Report the same decision-relevant estimates, units, and uncertainty measures in both analyses. Explain any difference in the populations or data supporting them.
Say whether the change affects the scientific interpretation or decision, and avoid claiming more robustness than the alternatives support.
A useful numerical comparison may include the estimate from each analysis, its uncertainty interval, the number of individuals and observations supporting it, and a change from the primary estimate. For a parameter \(\theta\), define the absolute change as \(\theta_{\mathrm{alt}}-\theta_{\mathrm{primary}}\). When the primary estimate is nonzero and the parameter scale makes ratios meaningful, define the relative change as:
The sign indicates direction: a negative value means the alternative estimate is lower than the primary estimate. Relative change can be misleading when the primary estimate is near zero, when estimates can be negative, or when the parameter is on a scale for which ratios lack a useful interpretation. In such cases, report the absolute change with units and explain its relevance.
Worked Examples of Reporting Sensitivity Results
Worked Example: Alternative Dataset and Clearance Estimate
In a hypothetical analysis, the primary dataset includes 120 participants and gives a typical apparent clearance estimate of 8.0 L/h, with a 95% confidence interval of 7.2 to 8.9 L/h. A planned sensitivity analysis excludes records from a data source with unresolved timing concerns; it includes 108 participants and estimates clearance at 8.3 L/h, with a 95% confidence interval of 7.4 to 9.4 L/h. Compare the estimates and describe the result without treating the intervals as a formal equivalence test.
The absolute change is \(8.3-8.0=0.3\) L/h. The relative change is:
The alternative estimate is 0.3 L/h, or 3.75%, higher than the primary estimate. Both intervals include values near the corresponding point estimates, and their ranges overlap substantially. This comparison supports the limited statement that the estimated typical clearance is similar under this particular exclusion. It does not prove the estimates are equivalent, and it does not show whether the excluded records were ignorable. The report should retain the change in sample size and explain the timing concern so readers can assess the relevance of the alternative.
Worked Example: Alternative Error Assumption and a Covariate Effect
Suppose a hypothetical primary analysis estimates a renal-function effect as a 1.22-fold change in clearance per specified covariate contrast, with a 95% confidence interval of 1.08 to 1.38. Under an alternative residual error assumption, the estimate is 1.19, with a 95% confidence interval of 1.03 to 1.37. The prespecified interpretation is that a factor of 1.25 or greater would be large enough to prompt consideration of a dosing adjustment. Determine what the comparison says about that decision threshold.
The point estimate changes by \(1.19-1.22=-0.03\). Its relative change is:
The point estimate remains below the 1.25 decision threshold under both assumptions, so the point-estimate-based decision does not change. However, the alternative interval extends to 1.37, above that threshold; the primary interval also extends above it, to 1.38. Thus, the estimates are similar, but uncertainty does not exclude effects larger than the threshold. A balanced report would state both facts. It would not claim that either analysis establishes the effect is too small to matter.
Worked Example: A Decision-Relevant Shift Despite Similar Uncertainty
In a hypothetical dose-selection evaluation, the primary analysis predicts a median exposure ratio of 1.18 for a subgroup relative to the reference group. An alternative analysis excluding participants with incomplete dosing records predicts 1.27. Assume the prespecified action threshold is 1.25, and that both analyses have similar uncertainty intervals: 1.10 to 1.27 for the primary result and 1.16 to 1.39 for the alternative. Calculate the shift and interpret its relevance.
The absolute shift is \(1.27-1.18=0.09\). Relative to the primary estimate:
The alternative point estimate crosses the 1.25 action threshold, whereas the primary point estimate does not. This is decision-relevant even though the shift is less than 8% and the intervals overlap. The result should be described as a change in the point-estimate-based conclusion, with substantial uncertainty in both analyses—not as proof that the alternative analysis is correct. The report should explain why records were excluded and whether that alternative dataset is appropriate for the intended question.
Justifying a Robustness Interpretation
The reasoning behind a robustness statement is a comparison of both numerical results and their implications. If a prespecified decision is represented by a threshold \(c\), and the primary and alternative estimates are \(\hat{\theta}_{\mathrm{primary}}\) and \(\hat{\theta}_{\mathrm{alt}}\), respectively, compare each with \(c\). If both lie on the same side, the point-estimate-based decision is unchanged; if they lie on opposite sides, it changes. That comparison does not by itself account for uncertainty, so the intervals and the consequences of uncertainty must also be described.
When the product is negative, the two point estimates are on opposite sides of the threshold. If either estimate equals the threshold, or if uncertainty spans it, the report should describe that boundary situation directly rather than forcing a binary conclusion. This simple check is useful for decision thresholds, but it does not replace scientific interpretation of the parameter, its units, or the uncertainty.
For each key result, the report should answer three questions: how much did the estimate change, how did its uncertainty change, and did the interpretation or decision change? A table is often effective when several analyses are compared, provided the text draws attention to consequential differences rather than leaving readers to infer them.
| Report element | What to include | Why it matters |
|---|---|---|
| Alternative definition | Changed assumption or records, rationale, and whether planned | Lets readers judge plausibility and reproducibility |
| Analysis basis | Participant and observation counts, and relevant data differences | Shows whether results rely on a changed evidence base |
| Key results | Primary and alternative estimates, units, and uncertainty | Enables direct comparison without obscuring precision |
| Decision consequence | Whether a threshold, interpretation, or conclusion changes | Connects numerical differences to the regulatory question |
Diagnostics, Alternatives, and Common Pitfalls
Before making a robustness claim, check that the comparisons are interpretable. Confirm that the same parameter definition, units, covariate contrast, and target population are being compared—or explicitly explain any change. Verify that each analysis actually changed the stated feature and that its sample size and data basis match the description. Check that uncertainty measures are identified consistently and that the report does not compare, for example, an estimate from one analysis with an interval from another.
If several choices change at once, the analysis may show that the combined alternative gives a different result, but it cannot identify which change caused the difference. Where the question matters, use separate sensitivity analyses that vary one feature at a time, or clearly label a combined alternative as such. A one-at-a-time analysis is easier to interpret but may not capture interactions between assumptions. The approach should match the question rather than imply that either design is universally superior.
Sensitivity analyses can also be compared with other ways of assessing uncertainty. The primary analysis remains the reference for the main reported result; a sensitivity analysis asks how results respond to specified alternatives. An uncertainty interval summarizes uncertainty within the assumptions of the analysis that produced it. These serve different purposes: an interval does not substitute for examining alternative assumptions, and a sensitivity analysis does not make the interval unnecessary.
A frequent mistake is to write, “The results were robust because the estimates were close.” Closeness alone has no universal meaning: a small numerical shift can cross an important decision threshold, while a larger shift may have no practical consequence. A second mistake is to infer equivalence from overlapping intervals. Overlap is not a formal test that differences are negligible. The correction is to state the estimates, uncertainty, threshold or decision context, and whether the conclusion changes, without asserting equivalence unless an appropriate equivalence framework was specified and used.
Other warning signs include reporting only alternatives that support the preferred conclusion, leaving the reason for an analysis unexplained, and calling an unplanned analysis “confirmatory.” Report the alternatives that are relevant to the question, distinguish planned from exploratory work, and make clear when an analysis is limited or inconclusive. The goal is transparent evaluation, not a vote among model runs.
Why the Reporting Matters
Regulatory readers need to understand whether the main conclusion depends on a fragile data choice or modeling assumption. A report that gives only a primary estimate cannot show that dependency; a long list of alternative estimates without interpretation cannot show its consequence. Effective reporting connects each plausible alternative to the same decision-focused comparison, while keeping the strength of the claim proportional to the evidence examined.
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- A primary estimate is 10.0 mg/L and an alternative is 10.5 mg/L. Calculate the absolute and relative changes. What additional information is needed before calling the result robust?
- Two analyses have overlapping uncertainty intervals, but their point estimates lie on opposite sides of a prespecified action threshold. Has the decision necessarily remained unchanged? Explain.
- An alternative analysis excludes 15 participants but also changes the residual error assumption. What limitation does this create for interpreting any difference?
- Why is it usually inappropriate to conclude that two estimates are equivalent simply because their uncertainty intervals overlap?
Solutions. (1) The absolute change is \(10.5-10.0=0.5\) mg/L; the relative change is \(100(0.5/10.0)=5.00\%\). Robustness cannot be judged without the uncertainty, rationale for the alternative, and decision context. (2) No. The point-estimate-based decision changes if the threshold separates the estimates; uncertainty and the decision framework must also be discussed. (3) The result reflects two changes together, so it cannot isolate whether the dataset or the error assumption produced the difference. (4) Interval overlap is not a test of negligible difference or equivalence; equivalence requires an appropriate prespecified framework and margin.