Why Test Nuisance Assumptions?
A group sequential design is planned using assumptions about the outcome distribution, variability, event rate, and other quantities that are not the primary treatment effect of interest. These nuisance parameters can strongly affect whether the trial has adequate power and how many participants it may need. A design that looks feasible under one plausible value may require substantially more enrollment under another.
Sensitivity analysis asks a direct planning question: if a nuisance assumption is different from its planning value, what changes? The aim is not to choose whichever scenario gives the most attractive result. It is to show which conclusions are stable, which depend on uncertain assumptions, and whether the design remains feasible across a defensible range.
A sensitivity analysis is informative only when its scenarios are tied to the design question. Changing the assumed standard deviation while keeping the enrollment cap fixed asks how the current design performs if variability differs from expectation. Recalculating the required enrollment to preserve a target information level asks a different question: how large would a revised design need to be? Report these questions separately.
Model and Assumptions for a Two-Look Example
Consider a hypothetical two-arm trial with a continuous outcome. Let \(N\) be the total number randomized, with equal allocation, and let \(\delta\) be the true difference in population means, in outcome units. Assume independent participants, normally distributed outcomes in both arms with a common known standard deviation \(\sigma\), complete outcome data, and no change in the target effect or analysis between looks. At total enrollment \(N\), the standard error of the difference in sample means is \(2\sigma/\sqrt{N}\). Thus the standardized statistic has variance 1 and, under a true difference \(\delta\), mean \(\delta\sqrt{N}/(2\sigma)\).
For the numerical illustration, the first analysis is at \(N_1=72\), and the maximum enrollment is \(N_{\max}=120\). The rule is to stop for efficacy if \(Z_1\geq 2.4\) at the first look; otherwise, stop for efficacy at the final look if \(Z_2\geq 2.0\). These are illustrative fixed boundaries. This example evaluates their behavior under different values of \(\sigma\); it does not establish that the boundaries control any particular overall type I error rate. In an actual design, global type I error must be checked under the null model.
Under this normal model, the two standardized statistics are jointly normal. Their means under the alternative are \(\delta\sqrt{N_1}/(2\sigma)\) and \(\delta\sqrt{N_{\max}}/(2\sigma)\), and their correlation is \(\sqrt{N_1/N_{\max}}\). The correlation reflects the information shared by the accumulated observations at the two looks. For the example it is \(\sqrt{72/120}=0.7746\).
Worked Example: Changing Variability With the Enrollment Cap Fixed
Suppose the target difference is \(\delta=4\) outcome units. Compare planning standard deviations of 8 and 10 outcome units. Keep the analysis looks, boundaries, and maximum enrollment of 120 fixed. The model assumptions above are being used as a transparent planning approximation; in practice, the common variance and complete-data assumptions should be checked against relevant prior evidence.
At the first look, the mean of \(Z_1\) under the target alternative is \(4\sqrt{72}/(2\sigma)\). For \(\sigma=8\), this is \(2.121\), so the chance of crossing the first boundary is:
For \(\sigma=10\), the corresponding mean is \(1.697\), and the first-look crossing probability is:
The larger standard deviation weakens the standardized signal, so early stopping is less likely. The maximum remains 120 in both scenarios because it was held fixed. Expected enrollment is 72 for trials stopping at the first look and 120 for trials continuing to the final look. Therefore:
Substituting the two crossing probabilities gives \(E(N)\approx120-48(0.390)=101.3\) participants when \(\sigma=8\), and \(E(N)\approx120-48(0.241)=108.4\) participants when \(\sigma=10\). These are expected enrollments under the specified rule, not maximum enrollments. The scenarios show a 7.16-participant increase in expected enrollment while the maximum stays at 120.
Power is the probability of crossing either boundary under the target alternative. For \(\sigma=8\), numerical evaluation of the joint-normal probabilities gives approximately 0.778. In full, the probability includes crossing at the first look or, conditional on not crossing then, crossing at the second look:
This is the power for the stated target difference, variability, enrollment schedule, and stopping rule. It is not a general power value for the boundary pair independent of the model.
How to Read and Build a Sensitivity Analysis
A useful sensitivity analysis begins by separating the quantities that are uncertain from the choices that define the design. For each scenario, state the nuisance value, the fixed design features, and the operating characteristics being compared. A compact scenario table makes these distinctions visible.
| Assumed standard deviation | Mean of \(Z_1\) under \(\delta=4\) | First-look crossing probability | Expected enrollment | Maximum enrollment |
|---|---|---|---|---|
| 8 outcome units | 2.121 | 0.390 | 101.3 participants | 120 participants |
| 10 outcome units | 1.697 | 0.241 | 108.4 participants | 120 participants |
The table changes only \(\sigma\). The assumed difference remains 4 outcome units, and the looks, boundaries, and maximum enrollment remain the same. This is a controlled comparison: the change in results can be attributed to the changed nuisance assumption under the specified model.
Choose scenario values using a rationale, such as historical estimates, external data, a range of expert opinion, or a conservative planning range. Do not imply that every value is equally likely unless a probability model justifies that claim. When several nuisance parameters are uncertain, one-at-a-time changes can reveal the effect of each parameter but miss interactions. A small, structured scenario matrix can show joint combinations—for example, higher variability together with a lower event rate—if those combinations are scientifically plausible.
Identify each nuisance parameter and explain why its planning value is uncertain.
Specify the range and its rationale before interpreting the results.
State whether the stopping rule, effect target, analysis schedule, and enrollment cap stay unchanged.
Examine power, error control, stopping probabilities, expected enrollment, and maximum enrollment as appropriate.
Compare the size of the change with prespecified feasibility or performance requirements, not with an arbitrary label such as “small.”
When the Enrollment Requirement Changes
The preceding example held the maximum enrollment fixed. A redesign can instead ask how maximum enrollment must change to preserve the same information under a different standard deviation. With equal allocation and known variance, the variance of the difference in sample means is \(4\sigma^2/N\). To keep this variance constant when \(\sigma\) changes, \(N\) must be proportional to \(\sigma^2\).
Worked Example: Recalculating the Maximum Enrollment
Suppose a hypothetical design has a maximum of 120 participants under \(\sigma_a=8\) outcome units. The planning team wants the same maximum information if the plausible standard deviation is instead \(\sigma_b=10\). Assume the same endpoint definition, equal allocation, and complete data; the calculation is a variance-based planning comparison, not a claim that all other design properties are unchanged.
The required maximum enrollment is:
Because enrollment must be an integer and equal allocation requires an even total, round up to 188 participants. This is 68 more than the original maximum. This result differs from the fixed-cap sensitivity example: there, the maximum stayed at 120 and expected enrollment changed; here, the maximum is deliberately revised to preserve the endpoint variance under a higher assumed standard deviation.
The quadratic relationship explains why variability can be a material planning assumption: a 25% increase in standard deviation leads to a 56.25% increase in the variance-matched sample size. The exact enrollment needed for a target power under a group sequential rule should be obtained by recalculating the operating characteristics with the revised information schedule and the intended boundaries, rather than relying on this scaling alone.
Diagnostics, Alternatives, and Limitations
A sensitivity result deserves scrutiny if it conflicts with the model’s direction or with basic design logic. With the effect held constant, increasing \(\sigma\) reduces the mean of the standardized statistic, so power and early efficacy stopping should not increase merely because variability rose. If they do, check scenario labels, units, the standard-error calculation, and whether the design was inadvertently changed at the same time. Also confirm that the reported maximum is a cap, not an average or an expected enrollment.
Sensitivity to nuisance assumptions is not the same as stress-testing operational deviations. This analysis varies assumed model quantities while holding the trial procedure fixed. An alternative approach is to redesign the trial separately for each scenario to meet a common objective; that answers how design requirements change, rather than how one fixed design performs. Report which approach was used. A fixed-design comparison and a scenario-specific redesign can both be useful, but their results answer different questions.
The main limitations are that a chosen range may omit important values, a simple model may not represent the data adequately, and sensitivity analysis cannot make an uncertain nuisance parameter known. The normal example assumes known common variance, independent outcomes, and complete data. If those assumptions are not credible, use a model and simulation procedure that reflect the planned analysis and relevant data structure. Likewise, examining power alone is insufficient when error control is a requirement: verify type I error under the null across the intended design scenarios.
A practical criterion for materiality should connect to decisions. For example, a team might define in advance that a scenario is consequential if it makes the target power unattainable under the enrollment cap, or if the required enrollment exceeds a recruitment limit. The reported conclusion should then name the assumption and the affected design quantity. “The design is robust” is too broad unless the relevant range, outcomes, and criteria are specified.
Check Your Understanding
Answer each question using the assumptions and distinctions developed above.
- If the assumed standard deviation increases while the effect, boundaries, and enrollment schedule stay fixed, what should generally happen to the mean of the standardized statistic under the alternative?
- In the two-look example, why is 108.4 participants not the maximum enrollment for the \(\sigma=10\) scenario?
- A design has a maximum of 160 participants at a standard deviation of 6 units. What maximum preserves the same endpoint variance at a standard deviation of 9 units, rounded up to an even total?
- What does a fixed-design sensitivity analysis answer that a scenario-specific redesign does not?
Solutions. (1) The mean \(\delta\sqrt{N}/(2\sigma)\) decreases when \(\sigma\) increases while the other quantities remain fixed. (2) It is the expected enrollment, averaging 72 participants for first-look stops and 120 for continuations; the fixed maximum is 120. (3) \(160(9/6)^2=160(2.25)=360\), already an even total, so the required maximum is 360 participants. (4) A fixed-design analysis shows how one design performs under alternative nuisance values; a scenario-specific redesign shows how the design requirements change when each scenario is used to plan a new design.