Why Test a Design Under Misspecification?
A design is usually evaluated by simulating data from a specified model and analyzing those data with a planned analysis pipeline. This is informative when the simulation model resembles the process that will generate future data. But the structure of that process is uncertain: a dose response may be nonlinear, residual variability may change with concentration, or measurements may have a different error distribution than assumed. A design that performs well only when the generating and analysis models match exactly may be fragile in practice.
Testing designs under model misspecification means deliberately generating data under plausible alternatives to the planned model, while retaining the planned analysis pipeline. The goal is not to prove that the planned model is wrong, nor to select the most complicated alternative. It is to learn whether a design’s performance changes materially when reasonable assumptions are perturbed.
The distinction between generating and analyzing models is essential. If data are generated from an alternative model but then analyzed with that same alternative, the exercise evaluates a different analysis plan. It does not show how the planned analysis will behave if its assumptions are violated.
Specify the Alternatives Before Simulating
A useful scenario changes one important assumption at a time when possible, so the source of a performance change is interpretable. Structural alternatives might add curvature, delayed response, or a different random-effects relationship. Error alternatives might change the variance function, introduce heavier tails, or alter the dependence between repeated measurements. Other plausible alternatives include covariate distributions or missingness mechanisms that differ from their planned assumptions.
State the target of estimation in each scenario. If the scientific target remains a parameter such as a population clearance, compare estimates with that parameter’s generating value. If the planned model has no parameter corresponding directly to the alternative mechanism, define a meaningful target in advance—for example, a specified exposure-response contrast. Without a clear target, “bias” can be ambiguous.
The design, simulated covariates and random effects, residual process, missing observations, and analysis pipeline should be specified consistently with the intended scenario. Replicates should be generated independently unless the simulation plan deliberately uses a paired construction for design comparisons. The analysis should use the same estimation and decision rules that would be used in the planned study, including convergence handling and interval construction.
Worked Example: A Design That Cannot Reveal Curvature
Suppose the planned analysis is a linear mean model in a scaled dose \(x\). Compare two designs using the same hypothetical structural alternative: \(m(x)=x+0.4x(x-1)(x-2)\), with independent residual errors distributed as \(N(0,0.2^2)\). The planned analysis still fits a straight line.
Design A samples at \(x=0,1,2\). Its true means are \(m(0)=0\), \(m(1)=1\), and \(m(2)=2\), because the product \(x(x-1)(x-2)\) is zero at each of those points. Thus, the means lie exactly on a straight line with slope 1. At these sample locations, the nonlinear alternative gives the linear analysis no mean-shape discrepancy to detect.
Design B also samples at \(x=0.5\) and \(x=1.5\). At \(x=0.5\), the true mean is \(0.5+0.4(0.5)(-0.5)(-1.5)=0.5+0.15=0.65\). At \(x=1.5\), it is \(1.5+0.4(1.5)(0.5)(-0.5)=1.5-0.15=1.35\). These values depart from the straight line \(m(x)=x\), even though the means at the original three locations do not.
The example isolates a design-sensitivity issue: sampling only at \(0,1,2\) makes this alternative look linear at the observed locations. Additional locations expose departures from linearity. This does not establish that Design B is always preferable; it shows how its sampling locations respond to this particular structural alternative. In a full simulation, replicate variability and the parameter or contrast of interest would also be evaluated.
Summarize Performance Within Each Scenario
For each scenario, calculate performance measures using the same definitions as in the primary design evaluation, but compare them with the scenario-specific target. Bias describes average direction and magnitude of estimation error. RMSE combines bias and variability: if all \(R\) replicates yield estimates, then \(RMSE=\sqrt{R^{-1}\sum_{r=1}^R(\hat{\theta}_r-\theta_s)^2}\). Coverage is the proportion of intervals that contain \(\theta_s\). A low coverage rate under an alternative error model indicates that the planned interval procedure may be miscalibrated in that scenario.
Report Monte Carlo uncertainty alongside simulation summaries when it matters for the decision. For example, an estimated coverage of 0.90 from 100 replicates is imprecise: its approximate Monte Carlo standard error is \(\sqrt{0.90(0.10)/100}=0.0300\). A difference in estimated coverage of a few percentage points should not be treated as a stable design distinction when simulation error is of similar size.
Worked Example: Coverage Under an Alternative Error Model
Consider a hypothetical design with 1,000 simulated data sets. The planned analysis constructs nominal 95% confidence intervals for a target parameter. Under the planned error model, 943 intervals contain the generating target. Under a plausible alternative with concentration-dependent residual variance, 812 contain it.
The estimated coverage under the planned model is \(943/1000=0.943\), or 94.3%. Under the alternative it is \(812/1000=0.812\), or 81.2%. The estimated reduction is \(0.943-0.812=0.131\), or 13.1 percentage points. These are simulation results for the stated hypothetical scenarios, not probabilities that a parameter lies in any particular interval.
For the alternative scenario, the approximate Monte Carlo standard error of the coverage estimate is \(\sqrt{0.812(1-0.812)/1000}=\sqrt{0.0001527}=0.0124\), or 1.24 percentage points. The shortfall from 95% is much larger than this Monte Carlo error, suggesting that the coverage problem is not merely simulation noise. Whether the design remains acceptable depends on the prespecified performance requirements and the consequences of undercoverage.
Compare Designs Without Hiding Their Weak Points
A design comparison should show performance by scenario, not only a single overall summary. A compact table can place each design’s bias, RMSE, coverage, and failure rate side by side for the nominal model and each alternative. This makes it possible to see, for example, that one design improves precision under the planned model but loses coverage under heteroscedastic errors.
Worked Example: Bias and RMSE Under a Structural Alternative
Suppose two hypothetical designs estimate a target \(\theta_s=1.00\) under one misspecified structural scenario. Each has 100 successful estimates and no failed fits. For Design A, the mean estimate is 0.96 and the mean squared estimation error is 0.220. For Design B, the mean estimate is 1.02 and the mean squared error is 0.160.
For Design A, the estimated bias is \(0.96-1.00=-0.040\). Its RMSE is \(\sqrt{0.220}=0.469\) (to three significant figures). For Design B, the estimated bias is \(1.02-1.00=0.020\), and its RMSE is \(\sqrt{0.160}=0.400\). In this scenario, Design B has smaller absolute bias and smaller RMSE. The bias values have the same units as \(\theta_s\); RMSE also has those units.
This comparison alone does not establish universal superiority. The result applies to the specified target, scenario, analysis pipeline, and simulation precision. If the designs have different performance in other plausible scenarios, present those results separately before making a broader design judgment.
Failure rates are also part of performance. If a model fails to converge in 40 of 1,000 replicates, the failure rate is \(40/1000=0.040\), or 4.0%. Report whether performance summaries exclude failed fits and how failures were defined. A design with apparently favorable conditional bias among successful fits may be unattractive if it frequently produces no usable estimate.
Diagnostics, Comparisons, and Limitations
Inspect more than summary averages. Performance distributions across replicates can reveal skewness, extreme estimates, multimodality, or a small group of severe failures hidden by a mean. Stratified diagnostics can identify whether sensitivity is concentrated in particular covariate or random-effects subgroups. Check that the simulated alternative actually produces the intended difference in data behavior, and that the planned analysis pipeline is being applied unchanged.
Three approaches answer different questions. Evaluating only under the planned model measures performance conditional on that model being correct. Re-fitting each scenario with its own correctly specified analysis can measure what might be achieved after changing the analysis, but does not test the robustness of the planned pipeline. Stress-testing alternatives with the planned analysis directly assesses sensitivity to misspecification. These results are complementary, not interchangeable.
Stress tests have limits. They cannot protect against alternatives that were never considered, and simulated results inherit any errors in the simulator. A severe alternative may be implausible, while a mild alternative may miss the clinically important failure mode. Specify why each scenario is plausible, use a range of meaningful perturbations where warranted, and avoid treating one scenario’s result as a general guarantee.
A common mistake is to simulate an alternative data-generating process and then silently analyze it with a model tailored to that alternative. The resulting favorable performance does not answer whether the planned model is robust. Keep the scenario generator and analysis pipeline explicit in the report. Also avoid declaring a design robust merely because its point estimate of bias is small: interval coverage, precision, failures, and simulation uncertainty may tell a different story.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A study is planned with a linear dose-response model. What must remain fixed when testing whether that analysis is sensitive to a nonlinear generating mechanism?
- A scenario has target \(\theta_s=2.0\), and the mean estimate across successful replicates is 1.9. What is the estimated bias?
- In 500 replicates, 450 nominal 95% intervals contain the target. Calculate the estimated coverage and its approximate Monte Carlo standard error.
- Why can a design with sampling only at \(x=0,1,2\) fail to reveal the curvature in the function \(m(x)=x+0.4x(x-1)(x-2)\)?
- Why should the number of failed fits be reported alongside bias and RMSE?
Solutions. (1) Keep the planned analysis model and its estimation, interval, and failure-handling pipeline fixed; change the data-generating mechanism to the nonlinear alternative. (2) The estimated bias is \(1.9-2.0=-0.1\) target units. (3) Coverage is \(450/500=0.900\), or 90.0%. Its approximate Monte Carlo standard error is \(\sqrt{0.900(0.100)/500}=\sqrt{0.000180}=0.0134\), or 1.34 percentage points. (4) At each of \(0,1,2\), the product \(x(x-1)(x-2)\) is zero, so the generated means are exactly \(0,1,2\), matching a straight line at those locations. (5) Bias and RMSE calculated only from successful fits can conceal that a design often fails to provide an estimate; the failure rate is part of its practical performance.