Why Stress-Test the Model, Not Just Its Parameters?
A design can perform as planned under its analysis model and still behave differently when trial outcomes depart from that model. For example, outcomes may be more skewed than assumed, their variance may be larger, or repeated measurements may be less strongly associated. Such departures can affect type I error, power, confidence-interval coverage, or the probability of reaching a design decision.
Sensitivity analysis for nuisance parameters varies quantities such as a standard deviation or a correlation within a chosen model. Model-misspecification stress-testing asks a broader question: what happens if the assumed form of the data-generating process is wrong? The distinction matters. Changing a variance while retaining a normal distribution is a parameter stress; replacing the normal distribution with a skewed distribution having the same mean and variance is a shape stress.
For a candidate design, first specify the model used to analyze the trial and the data-generating assumptions used to plan it. Then create alternative data-generating scenarios that depart from those assumptions in scientifically credible ways. Apply the same planned analysis and decision rule to every scenario. The resulting operating characteristics show how the design-analysis pair performs when reality differs from the planning model.
State the Assumptions Before Stressing Them
A stress test is interpretable only when the reference model and the departure are explicit. Consider a two-arm trial analyzed by comparing treatment and control means. A simple planning model might assume independent participants, normally distributed outcomes within each arm, a common variance, and a prespecified two-sided test at level \(\alpha=0.05\). Its data-generating model also specifies the mean difference and variance.
These assumptions are not all the same kind. The mean difference and variance are parameters; normality, independence, and a particular covariance structure describe model form. A scenario that changes only the variance tests sensitivity to that parameter. A scenario that changes skewness while holding the mean and variance fixed tests a distributional assumption. A scenario that changes correlation among repeated measurements tests the assumed association structure.
In an adaptive design, each simulated replicate must run through the full planned procedure: generate trial data, apply the interim and final analyses, and record the resulting decision or estimate. This tutorial focuses on outcome-model departures. Other design features should remain fixed unless they are themselves the departure under study; changing several features at once can make the cause of a performance change unclear.
For each scenario, report the target operating characteristic and its Monte Carlo uncertainty. If \(R\) independent simulated trials yield \(K\) rejections, the estimated rejection probability is \(\hat{p}=K/R\), with estimated Monte Carlo standard error \(SE=\sqrt{\hat{p}(1-\hat{p})/R}\). This standard error quantifies simulation error, not uncertainty about whether the scenario is realistic.
Variance Departure: A Direct Calculation
A simple normal approximation shows how a variance departure can change rejection probability even when the planned test and effect are unchanged. Suppose the planned statistic is compared with the two-sided normal critical values \(-1.96\) and \(1.96\). Under a particular alternative, assume its planned distribution is approximately normal with mean \(2.1213\) and standard deviation 1. If the actual standard deviation is 1.2 instead, the statistic has more spread than anticipated.
Worked Example: Power Under Variance Inflation
Hypothetical setup. A trial uses a two-sided test with \(\alpha=0.05\), rejecting when \(|Z|>1.96\). Under the alternative, the planned model gives \(Z\sim N(2.1213,1)\). A stress scenario retains the mean but inflates the standard deviation to 1.2, so \(Z\sim N(2.1213,1.2^2)\). The normal approximation and the stated distributions are assumed to hold for this calculation.
Under the planning model, the rejection probability is
Here \(\Phi\) is the standard normal cumulative distribution function. Under the variance stress, standardizing by the actual standard deviation gives
The stress scenario reduces rejection probability from about 0.5641 to 0.5538, a decrease of about 0.0103, or 1.03 percentage points. The effect is not simply a uniform reduction: increased spread can also increase the probability of crossing the lower critical boundary, although that contribution here is small. This calculation illustrates a single stress, not a general formula for all trial designs.
Distributional Shape: Preserve Moments, Change the Form
A useful shape stress changes the distribution while keeping its mean and variance fixed. That isolates consequences of shape more clearly than changing the mean, variance, and skewness simultaneously. For example, a standardized normal outcome and a standardized exponential-derived outcome can both have mean 0 and variance 1, while the latter is strongly right-skewed.
The operating characteristics under these scenarios should be evaluated using the analysis that would actually be used in the trial. If that analysis relies on a normal approximation or a small-sample model, the simulation tests whether its behavior remains acceptable under the departure. Do not silently replace the planned analysis with one that is better suited to the stress scenario: doing so answers a different question. A separate robust analysis can be compared deliberately, with the change documented.
Worked Example: Same Moments, Different Outcome Shape
Hypothetical setup. Consider a two-arm trial with 30 participants per arm and a two-sided pooled \(t\)-test at \(\alpha=0.05\). The planned model assumes independent normal outcomes with a common variance. For a shape stress, generate control outcomes from \(G-1\), where \(G\) has an exponential distribution with mean 1; generate treatment outcomes from \(0.65+G-1\), independently. Both groups have variance 1, and the mean difference is 0.65, but the outcomes are right-skewed rather than normal. The two simulation scenarios use the same sample sizes, effect, variance, analysis, and decision rule; only outcome shape differs.
Suppose 5,000 independent simulated trials under the normal planning model yield 3,710 rejections. The estimated power is
Suppose 5,000 simulated trials under the skewed stress scenario yield 3,545 rejections. Then
For independent simulation runs, the estimated difference in power is \(0.742-0.709=0.033\). Its Monte Carlo standard error is approximately
The estimated difference is about 3.70 Monte Carlo standard errors. A rough 95% Monte Carlo interval for this difference is \(0.033\pm1.96(0.00892)\), or approximately \([0.0155,0.0505]\). Thus the simulation suggests a lower rejection probability under the skewed scenario, and the difference is larger than would be expected from Monte Carlo noise alone. These counts are illustrative hypothetical simulation results, not empirical trial findings. A design team would also examine type I error under the corresponding null scenarios; power alone does not establish acceptable performance.
Association Departure: Repeated Measurements
An assumed correlation can affect precision even when each measurement has the correct marginal mean and variance. For two measurements on the same participant, with standard deviations \(\sigma_1\) and \(\sigma_2\) and correlation \(\rho\), the variance of their difference is
This follows by expanding the variance of a difference: \(\operatorname{Var}(Y_2-Y_1)=\operatorname{Var}(Y_2)+\operatorname{Var}(Y_1)-2\operatorname{Cov}(Y_1,Y_2)\), and using \(\operatorname{Cov}(Y_1,Y_2)=\rho\sigma_1\sigma_2\). Thus, for positive correlation, assuming a correlation that is too high can understate the variability of change scores and overstate precision.
Worked Example: Lower Correlation, Less Power
Hypothetical setup. A trial compares mean change between two equally sized arms, with 40 participants per arm. At baseline and follow-up, each measurement has standard deviation 10 points. The treatment-control difference in mean change is 5 points. Assume independent participants and arms, known variance, and a normal approximation for the test statistic. The planning correlation between baseline and follow-up is 0.60, but a stress scenario uses 0.20.
Under the planning correlation, the change-score variance for one participant is
The standard error of the between-arm difference in mean change is therefore
The test statistic has mean \(5/2=2.500\) under the alternative and standard deviation 1 under the assumed known-variance normal model. Its two-sided rejection probability is approximately \(\Phi(2.500-1.96)+\Phi(-1.96-2.500)=0.7054\), or about 0.705.
With the stress correlation of 0.20, the change-score variance becomes
The planned known-variance test continues to divide by its planned standard error of 2. Under the stress scenario, the statistic therefore has mean \(5/2=2.500\) and standard deviation \(\sqrt{160/40+160/40}/2=\sqrt{2}\). Its rejection probability is
Under these assumptions, the lower correlation reduces power from about 0.705 to 0.6495. The calculation isolates the effect of correlation on change-score variance. In a real analysis, the variance would generally be estimated, so simulation should include that estimation step rather than treating it as known.
A Practical Stress-Testing Workflow
Record the outcome distribution, means, variances, dependence structure, analysis, and decision rule used for the planned operating-characteristic calculation.
Specify whether the scenario changes shape, variance, association, or another assumption. Preserve other features where feasible so the effect of the departure is interpretable.
For every replicate, generate data under the scenario and apply the exact analysis and decision process intended for use. Record a clear outcome such as rejection, coverage, or a selected design decision.
Report the estimated operating characteristic, Monte Carlo standard error or interval, and the corresponding reference-model result. Keep simulation error distinct from uncertainty about the scenario itself.
Check that results are not caused by coding errors, a mismatched analysis, or inadequate simulation precision. Then decide whether the design is robust enough or whether assumptions, sample size, or analysis require reconsideration.
Diagnostics, Comparisons, and Limitations
A stress test can fail to answer its question if the simulated departure is not what was intended. Inspect generated data summaries and plots: means, variances, tail behavior, and empirical correlations should match the scenario specification. Check that the planned analysis is applied unchanged and that null scenarios are included when type I error control matters. Repeating the simulation with more replicates can reduce Monte Carlo error, but it cannot repair an implausible scenario or a misspecified simulation generator.
Model-based calculations are useful for transparent checks, as in the variance and correlation examples. Simulation is more flexible when outcomes are skewed, procedures are adaptive, or nuisance parameters are estimated. A robust analysis may be less sensitive to some departures, but it is not automatically superior: it can lose efficiency under the reference model, and it too has assumptions. A fair comparison applies each candidate analysis across the same scenarios and compares the relevant operating characteristics.
Stress-testing has limits. It cannot prove that a design will perform well under every possible data-generating process. Results depend on which departures are chosen and how severe they are. A scenario that changes several assumptions at once may be useful as a broad severe test, but it does not identify which assumption drives the result. Conversely, changing one feature at a time may miss interactions among departures. A transparent evaluation therefore includes a defensible set of focused scenarios, with selected combined scenarios when scientifically plausible.
Common mistake: “The design achieved its target power under the planning model, so it is adequately powered.” This conclusion ignores the possibility that the assumed distribution or association structure is wrong. The correction is to report performance under the reference model and under prespecified stress scenarios, alongside Monte Carlo uncertainty. A low stress-scenario estimate is evidence about that scenario, not proof that the trial will have that exact power.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- What is the difference between increasing an outcome variance within a normal model and replacing a normal outcome with a skewed outcome of the same variance?
- A simulation estimates a rejection probability of 0.60 from 2,000 independent trials. Calculate its Monte Carlo standard error.
- Why can a higher assumed baseline-follow-up correlation lead to an overoptimistic precision estimate for a change-score analysis?
- In a shape stress test, why should the planned analysis usually be applied unchanged?
- Does increasing the number of simulated trials resolve uncertainty about whether a stress scenario is scientifically plausible? Explain.
Solutions. (1) The first changes a parameter, the variance, while retaining the distributional form; the second changes model form while holding the specified variance fixed. (2) \(SE=\sqrt{0.60(0.40)/2000}=\sqrt{0.00012}\approx0.01095\), or about 0.0110. (3) Since \(\operatorname{Var}(Y_2-Y_1)=\sigma_2^2+\sigma_1^2-2\rho\sigma_1\sigma_2\), a higher positive correlation reduces the assumed change variance and hence the estimated standard error. (4) Keeping the analysis fixed tests whether the intended design-analysis procedure remains reliable under the changed data-generating process; changing it would evaluate a different procedure. (5) No. More simulated trials reduce Monte Carlo error, but scenario plausibility requires scientific and operational judgment.