What Makes a Plan Fragile?
A sample-size calculation can be arithmetically correct and still support a fragile plan. Fragility arises when a small, uncertain, or weakly justified change in an input produces a large change in the required enrollment, the power, or the interpretation of the endpoint. The problem is not uncertainty by itself: planning always involves uncertain inputs. The problem is relying on an assumption without establishing that it is credible, examining its consequences, or deciding what to do if it fails.
The earlier material on sensitivity analysis for uncertain sample-size inputs shows how to vary planning inputs. Here the emphasis is diagnostic: identifying which assumptions deserve scrutiny, checking whether they are supported, and choosing a corrective analysis that addresses the source of risk. A robust plan should distinguish what is supported by data, what is a modeling assumption, and what is a decision made for clinical or operational reasons.
A useful diagnosis links each assumption to observable evidence and a consequence. For example, a variance estimate from a small pilot may have substantial uncertainty; the consequence is that the planned sample size may be too small if the true variance is higher. A threshold chosen after looking at treatment results may exaggerate the apparent responder effect; the consequence is that the planned endpoint may not provide a reliable confirmatory test. These are distinct problems and require different checks.
Check Effect-Size Credibility
For a two-arm comparison of a continuous outcome, a common planning approximation assumes independent normally distributed outcomes with a common, known standard deviation \(\sigma\), equal allocation, a prespecified two-sided type I error rate \(\alpha\), and a target power \(1-\beta\). If the treatment contrast to be detected is \(\Delta\), the approximate sample size per arm is
This is a planning approximation, not a guarantee of realized power. Its key diagnostic implication is that \(n\) is proportional to \(1/\Delta^2\): halving the assumed effect requires four times the sample size, if the other inputs remain fixed. An effect selected because it makes enrollment feasible is not thereby a credible effect. Check whether it is compatible with the clinical objective, prior evidence, the endpoint scale, and the treatment contrast the study can reasonably deliver.
Worked Example: An Attractive but Optimistic Effect
Consider a hypothetical two-arm trial with a continuous week-12 endpoint. Suppose the planning assumptions are a common standard deviation of 9.6 points, two-sided \(\alpha=0.05\), 80% power, and equal allocation. The proposed effect to detect is 4.0 points. The normal quantiles are approximately \(z_{0.975}=1.96\) and \(z_{0.80}=0.8416\). The planning formula gives:
Rounding up gives 91 participants per arm for the analyzable sample under these assumptions. Suppose, however, that the clinically important effect is 2.8 points and available evidence does not support assuming an effect as large as 4.0 points. Using 2.8 points instead:
so the target becomes 185 analyzable participants per arm. With a hypothetical 15% loss of analyzable outcomes, the enrollment targets would be \(\lceil91/0.85\rceil=108\) and \(\lceil185/0.85\rceil=218\) per arm, respectively. The diagnostic conclusion is not that the larger target must be chosen automatically. It is that feasibility changes substantially with the effect assumption, so the effect must be justified rather than selected to fit the first target. Attrition inflation also assumes the stated loss proportion is appropriate and does not replace analysis of how missing outcomes may affect the estimand or inference.
A corrective analysis can present a range of sample sizes across clinically credible effects, identify the effect at which enrollment becomes infeasible, and explain the implications for the development decision. External evidence should be relevant to the population, dose, endpoint, and follow-up period; a treatment effect from a different setting is not automatically transportable.
Check Variance Stability and Model Assumptions
An estimated standard deviation can be unstable when based on a small or unrepresentative sample. Because the continuous-outcome formula depends on \(\sigma^2\), a 20% increase in the assumed standard deviation raises the required sample size by \(1.2^2=1.44\), or 44%, with the effect and other assumptions unchanged. Check the source, sample size, population, and measurement conditions behind the variance estimate. In longitudinal designs, also check whether the planned correlation structure is supported: the precision of a change score depends on the variability at each time and their correlation.
Worked Example: An Uncertain Standard Deviation
Suppose a hypothetical pilot study suggests a standard deviation of 8 points for the outcome. For a two-arm design with \(\alpha=0.05\), 80% power, and a target effect of 3 points, the same approximation gives:
or 112 analyzable participants per arm. If the pilot is small and a plausible higher standard deviation is 10.5 points, then:
or 193 per arm. The ratio of the unrounded sample sizes is approximately \((10.5/8)^2=1.7227\). The large difference signals that treating the pilot estimate as known would be risky. A focused response is to obtain a more relevant variance estimate if possible, show a planning range, or choose a design that can meet its objective under a defensible conservative variance. The hypothetical calculation does not establish that 10.5 is the correct standard deviation; it shows the consequence if the lower estimate is wrong.
For repeated measurements, an additional fragility may lie in an assumed within-person correlation. For two measurements with equal standard deviation \(\sigma\) and correlation \(\rho\), the variance of an individual change is \(2\sigma^2(1-\rho)\). This follows by expanding the variance of the difference: \(\operatorname{Var}(Y_2-Y_1)=\operatorname{Var}(Y_2)+\operatorname{Var}(Y_1)-2\operatorname{Cov}(Y_1,Y_2)\). Thus, a high assumed correlation reduces the calculated variability of change and may make the planned sample size too small if the actual correlation is lower.
Worked Example: A Correlation Assumption Drives Enrollment
Suppose a hypothetical longitudinal trial plans to compare mean change between two equally sized arms. Assume independent participants, equal standard deviations of 10 points at baseline and follow-up, a common within-person correlation, a target change contrast of 4 points, two-sided \(\alpha=0.05\), and 80% power. Under a correlation of 0.60, the variance of change is \(2(10^2)(1-0.60)=80\). The approximate sample size per arm for a two-group difference in mean change is:
or 79 analyzable participants per arm. If the correlation is instead 0.20, the change variance is \(2(10^2)(1-0.20)=160\), giving:
or 157 per arm. The calculation assumes that the stated correlation and standard deviations characterize the planned population and that the normal approximation is adequate. The two-fold difference in variance makes the correlation assumption consequential. Examine relevant longitudinal data and assess a credible range of correlations; do not infer a precise correlation from a handful of observations without acknowledging its uncertainty.
Test Whether the Endpoint Is Well Supported
An endpoint must be defined sufficiently clearly to determine what is measured, when it is measured, how change is summarized, and how intercurrent events and missing observations are handled. A responder endpoint adds a threshold, so its clinical interpretation and evidential basis should be explicit. Check that the threshold is supported independently of the treatment-effect results, that the measurement scale can meaningfully distinguish values around the threshold, and that the threshold can be applied consistently across participants.
Worked Example: Sensitivity to a Responder Threshold
In a hypothetical study, let improvement be a score where larger values are better. Suppose, for illustration, improvement is normally distributed with standard deviation 3 points in both arms, with means 6 points under treatment and 3 points under control. A responder is defined as having improvement of at least 5 points. Under these distributional assumptions, the response probabilities are:
The modeled difference in response probabilities is approximately \(0.6306-0.2525=0.3781\), or 37.81 percentage points. At a threshold of 7 points, the probabilities are approximately 0.3694 and 0.09121, a difference of 0.27823, or 27.82 percentage points. This change shows why threshold selection can affect interpretation and planning. The values are model-based illustrations, not empirical estimates; a real analysis should examine the observed score distribution and justify its threshold clinically and methodologically. If conclusions change materially across plausible thresholds, report that dependence and avoid presenting one threshold as uniquely established without supporting evidence.
When the definition is weakly supported, corrective work may include reviewing measurement properties and clinical interpretation, prespecifying a defensible threshold, and examining a continuous endpoint alongside the responder analysis when scientifically appropriate. A continuous analysis is not a substitute for justifying a responder definition if the responder outcome is the planned primary endpoint.
Target Diagnostics and Corrective Analyses
Model assumptions should be examined in the form relevant to the design and endpoint. For a longitudinal mixed model, for example, inspect whether the planned mean structure captures the treatment-by-time contrast of interest and whether the assumed covariance structure is compatible with available data. For a binary or time-to-event endpoint, examine whether event-rate or event-time assumptions are supported in the relevant population and follow-up period. A diagnostic should answer a specific question; generic requests to “check the model” do not identify the failure or its consequence.
Record the assumed effect, variance, endpoint rule, correlation, event rate, or model structure, and state why it was chosen.
Check its source, relevance to the intended population and design, and whether the data supporting it are sufficiently informative.
Vary the uncertain assumption over a credible range and quantify changes in enrollment, precision, endpoint interpretation, or operating characteristics.
Strengthen the evidence, revise the endpoint or model, increase enrollment if feasible, or state clearly which conclusions depend on the assumption.
The main alternatives to a single-input plan are complementary rather than interchangeable. A scenario analysis shows consequences across specified inputs but does not assign probabilities to those scenarios. A simulation can assess a complete design under multiple data-generating conditions, including model departures, missingness, and analysis procedures, but its conclusions depend on the scenarios simulated. A more conservative planning value can protect against one identified uncertainty, but may be inefficient if it is not credible or if another assumption is the true source of risk. The choice should follow the diagnosis.
Common warning signs include a target effect close to the largest value suggested by available evidence, a variance taken from a small or mismatched study, a threshold selected after examining outcomes, or a repeated-measure correlation assumed without supporting data. Also look for abrupt changes in required enrollment under modest input changes, model fits or residual patterns inconsistent with assumptions, and sensitivity results that are reported without a clear account of which assumptions are plausible. No single diagnostic eliminates uncertainty. Sensitivity analyses are limited by the scenarios considered, and simulations do not rescue an endpoint whose clinical meaning is unclear.
Common pitfall: “The calculation gives the desired power, so the plan is adequate.” This reasoning treats the assumed effect, variance, endpoint, and model as if they were established facts. The correction is to verify the calculation and separately assess the credibility and uncertainty of each consequential assumption. A numerically precise sample size can still be scientifically fragile.
Check Your Understanding
Use the assumptions stated in each question and give a brief justification.
- If the target effect in a two-arm continuous-outcome plan is halved while all other inputs stay fixed, by what factor does the approximate sample size change?
- A plan assumes a standard deviation of 7 points, but a credible sensitivity value is 9 points. What is the ratio of the required sample sizes under the two values?
- Why does selecting a responder threshold after looking at treatment outcomes make the endpoint plan fragile?
- In a repeated-measure design with equal measurement variances, does reducing the assumed within-person correlation increase or decrease the variance of change? Explain.
Solutions. (1) It increases by a factor of 4 because the sample size is proportional to \(1/\Delta^2\). (2) The ratio is \((9/7)^2=81/49\approx1.653\), so the 9-point standard deviation requires about 1.65 times the sample size, before rounding. (3) Choosing the threshold using the observed results can make the apparent treatment difference depend on a data-driven definition, weakening the credibility and reproducibility of the planned endpoint; the threshold should have independent justification and be prespecified. (4) It increases: for equal standard deviations \(\sigma\), the change variance is \(2\sigma^2(1-\rho)\), which is larger when \(\rho\) is smaller.