Why Plan Sensitivity Analyses?
A pharmacometric analysis depends on assumptions that may be reasonable but not certain: recorded administration times may be imperfect, assay results may have a systematic calibration difference, or missing observations may differ from what the model predicts. A primary analysis must make decisions about such assumptions. A sensitivity analysis asks a focused follow-up question: if a specified assumption were changed in a plausible way, how much would the result and its interpretation change?
Planning the analysis in advance makes the answer interpretable. The alternative should challenge a named assumption, be feasible with available data or a clearly stated scenario, and preserve the scientific objective. The comparison should also be defined before seeing the alternative result. Otherwise, it is easy to choose whichever reanalysis produces the most appealing estimate.
Specify the Assumption, Alternative, and Comparison
A useful plan starts with the primary analysis as the reference. For every sensitivity analysis, name (1) the assumption being challenged, (2) why a different assumption is plausible, (3) the exact alternative analysis, and (4) how its result will be compared with the reference. Keep the objective and parameter definition fixed wherever possible. If the alternative changes the target population, parameter, or decision rule, a difference in results may not be attributable to the assumption being challenged.
Prioritize assumptions that are uncertain and could plausibly affect an estimate or decision. Do not list every imaginable variation.
State the data source, transformation, scenario, or model constraint that will change. Make the rule reproducible rather than saying “use a reasonable alternative.”
Use the same objective, analysis population, and decision criteria unless one of these is itself the assumption under investigation. State any necessary exceptions.
Choose relevant parameters, units, a measure of change, and any clinical or scientific boundary that determines whether the conclusion changes.
For a positive estimate \(\theta_{\mathrm{ref}}\) in the reference analysis and \(\theta_{\mathrm{alt}}\) in an alternative analysis, a convenient signed relative change is
The sign shows the direction of change; the magnitude describes its size relative to the reference. This measure is not suitable when the reference estimate is zero or very close to zero. In that case, prespecify an absolute difference in the parameter’s units, or a scale appropriate to the parameter. In either case, report the estimates themselves as well as the change: a percentage alone can hide whether the difference matters.
There is no universal cutoff for a “material” change. A threshold should be justified by the scientific objective, clinically relevant scale, or a prespecified decision rule. It should not be selected after observing which cutoff makes the results appear robust. Compare conclusions separately from estimates: two estimates can differ while remaining on the same side of a decision boundary, or differ only modestly yet fall on opposite sides.
Worked Examples of Planned Comparisons
Worked Example: Uncertain Administration Times
Hypothetical setting. A population PK analysis uses administration times transcribed from a clinical record. The primary analysis assumes those times are accurate to the recorded minute. Source records suggest that some times may instead have been rounded to the nearest hour. The objective is to estimate clearance, \(CL\), and assess whether the typical steady-state exposure for a 100 mg dose exceeds a prespecified target AUC of 18 mg·h/L.
Plan. The alternative uses source-verified administration times wherever available and applies a prespecified reconstruction rule to records without a verified time. The same subjects, concentration records, model, and exposure target are retained. The plan records how many times were replaced and the reconstruction rule; it does not permit choosing a different time separately for each subject after inspecting model results.
Suppose the reference estimate is \(CL_{\mathrm{ref}}=4.80\) L/h and the alternative estimate is \(CL_{\mathrm{alt}}=5.10\) L/h. The signed relative change is
For this illustrative calculation, assume linear dose proportionality and use \(AUC= Dose/CL\). The reference exposure is \(100\text{ mg}/(4.80\text{ L/h})=20.83\) mg·h/L. The alternative exposure is \(100\text{ mg}/(5.10\text{ L/h})=19.61\) mg·h/L. Its signed relative change is \(100(19.61-20.83)/20.83=-5.86\%\), with the small rounding difference due to displayed values.
Both estimates exceed the prespecified target of 18 mg·h/L. Thus the clearance estimate changes by 6.25%, while the stated target comparison remains the same. This result would support robustness of that particular conclusion to the timing alternative; it would not prove that all recorded times are accurate or that other conclusions are robust.
Worked Example: Assay Calibration Difference
Hypothetical setting. Concentrations from two assay lots are combined in the primary analysis, which assumes that the reported values are on a common calibrated scale. A validation exercise indicates that one lot may read approximately 6% low. The primary target is typical trough concentration, in mg/L.
Plan. Use the primary reported concentrations in the reference analysis. In the alternative, multiply concentrations from the affected lot by the prespecified correction factor \(1/0.94\), while leaving other records unchanged, and refit the same model. This analysis tests the consequence of the stated calibration concern; it does not independently validate the correction factor.
Suppose typical trough concentration is 1.20 mg/L in the reference analysis and 1.26 mg/L in the alternative. Then
Suppose the prespecified interpretation is whether typical trough concentration is at least 1.00 mg/L. Both estimates exceed that boundary: the reference estimate by \(1.20-1.00=0.20\) mg/L and the alternative by \(1.26-1.00=0.26\) mg/L. The estimate shifts upward by 0.06 mg/L, but this threshold-based conclusion is unchanged. The plan should report the lot-specific correction and its basis so that the alternative can be reproduced.
Worked Example: A Plausible Scenario for Missing Follow-Up
Hypothetical setting. A pharmacodynamic analysis estimates a treatment response, and the primary model assumes that missing follow-up responses are adequately represented by predictions conditional on the observed data. Investigators consider it plausible that participants who discontinue may have lower subsequent responses than those predictions. This is an assumption to challenge, not a fact established by the observed records.
Plan. Prespecify a scenario in which each unobserved post-discontinuation response is 15% below its corresponding primary-model prediction, then refit the analysis using that scenario. Keep the target population, outcome scale, and decision boundary fixed. The scenario’s timing and the exact response values it affects must be specified in the analysis plan.
Suppose the reference estimate of mean response is 24.0 units and the scenario estimate is 21.6 units. The absolute change is \(21.6-24.0=-2.4\) units, and the signed relative change is
If the prespecified decision boundary is a mean response of 22.0 units, the reference estimate is above it by \(24.0-22.0=2.0\) units, whereas the scenario estimate is below it by \(21.6-22.0=-0.4\) units. The decision-rule classification therefore changes under this scenario. The correct interpretation is that the conclusion is sensitive to the specified missing-response assumption. This analysis cannot show that the scenario is true, and a complete-case analysis alone would not resolve the underlying uncertainty.
Justifying the Comparison and Reading the Results
The comparison is meaningful only when it connects the changed assumption to the original objective. A short justification helps. If \(g(\theta)\) is the decision quantity derived from a model parameter \(\theta\), then comparing \(g(\theta_{\mathrm{ref}})\) with \(g(\theta_{\mathrm{alt}})\) is useful when the decision is made on the \(g\)-scale. For example, a clearance change is not automatically a clinically important exposure change. In the first example, the model-based transformation \(AUC=Dose/CL\) translates the clearance estimates into the exposure measure used by the target rule. The assumptions behind that transformation must themselves be reasonable for the stated calculation.
A plan should distinguish at least three outcomes of a sensitivity analysis:
- Estimate stability: the estimate changes little on a scale judged relevant for the objective.
- Conclusion stability: a prespecified interpretation or decision rule is unchanged.
- Fragility: an estimate changes materially, or a decision classification changes, under a plausible alternative.
These outcomes are not interchangeable. A decision can remain stable despite a noticeable shift in an estimate, as in the first example. Conversely, a modest estimate change can matter if the reference result lies close to a decision boundary. Sensitivity analyses are also not hypothesis tests of which assumption is correct. A small difference does not prove the primary assumption, and a large difference does not by itself identify the better analysis.
Diagnostics, Alternatives, and Limitations
Before interpreting a difference, check that the alternative analysis actually ran as planned. Verify the records changed, the correction or scenario was applied only where specified, the intended analysis population and target were retained, and the model produced estimates on the same scale. Review convergence, parameter boundaries, implausible predictions, and whether the alternative created data or model conditions that make the estimate uninterpretable. A numerical result from a failed or materially altered fit is not a valid robustness comparison.
| Approach | What it answers | Important limitation |
|---|---|---|
| One-at-a-time sensitivity analysis | How much the result changes when one named assumption changes | May miss interactions between assumptions |
| Combined alternative scenario | How results behave when several plausible changes occur together | A changed result may be hard to attribute to any single change |
| Exploratory reanalysis | What patterns or potential problems may warrant further investigation | Choices made after seeing results do not provide the same prespecified robustness assessment |
A strong plan generally begins with targeted, interpretable alternatives. If two assumptions could interact, a combined scenario may be useful, but it should be labelled as such and not substituted for understanding the separate changes. Sensitivity analyses also have limits: they cannot repair poor data, make an implausible alternative informative, or quantify all sources of uncertainty. If a relevant assumption has no credible alternative supported by data or a defensible scenario, state that limitation rather than implying it was tested.
Common Planning Mistakes
- Changing several things without saying so. If the objective, population, data rules, and model all change together, the cause of a result difference is unclear. Specify each change and justify any combined scenario.
- Using an undefined alternative. “Use corrected times” is not reproducible. State the source, correction, and rule for records lacking a verified value.
- Calling a result robust because it is not statistically significant. A significance test does not establish that two estimates are sufficiently close. Compare prespecified estimates, units, and decision criteria instead.
- Choosing the materiality threshold after seeing results. This makes the robustness claim dependent on the observed difference. Set a context-based threshold or decision boundary in advance.
- Treating the alternative as the truth. A sensitivity scenario explores consequences of an assumption; it does not establish that the alternative mechanism occurred.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A reference estimate is 8.0 mg/L and an alternative estimate is 8.4 mg/L. Calculate the signed relative change.
- Why should an alternative specify a rule for changing records rather than say “use plausible values”?
- A reference estimate is 19 units and an alternative is 17 units. The decision boundary is 18 units. Does the decision-rule classification remain stable?
- What can be concluded if a sensitivity analysis changes the estimate substantially?
- Why might a one-at-a-time analysis be insufficient when two assumptions could interact?
Solutions. (1) \(100(8.4-8.0)/8.0=5.00\%\). (2) A specific rule makes the analysis reproducible and helps ensure differences are due to the intended assumption rather than discretionary choices. (3) No. The reference is above 18 units and the alternative is below it, so the classification changes. (4) The result is sensitive to the tested assumption; the analysis alone does not establish which assumption is true. (5) The joint effect may differ from either separate effect, so a prespecified combined scenario may be needed to assess that interaction.