Why Stress-Test Structure?
A trial simulation can be internally consistent and still produce conclusions that depend strongly on an unverified model choice. A logit link rather than a probit link, a delayed rather than immediate treatment effect, or a curved rather than linear exposure-response relation can change predicted response rates and the probability of meeting a decision criterion. Stress-testing asks whether the trial conclusion is stable across scientifically plausible versions of these structures.
This is different from varying uncertain parameter values within one model. Parameter uncertainty asks how results change when, for example, an estimated treatment effect varies while the model form is held fixed. Structural sensitivity analysis changes the form of the model itself. It complements parameter uncertainty and the earlier material on detecting bias from model misspecification; it does not establish which candidate structure is true.
A useful comparison starts with a shared trial design, population, randomization, endpoint definition, and decision rule. Change a clearly identified structural feature, then examine the same outputs under each version. Keep common inputs fixed unless the changed structure requires a recalibration. When feasible, use the same simulated participants and random-number streams across scenarios. This paired comparison can make differences easier to interpret because it reduces variation due merely to different simulated samples.
Define the Comparison Before Simulating
A candidate structure is not a stress test simply because it is mathematically different. Each alternative needs a scientific interpretation and a clear statement of what is held constant. For example, compare time-course forms while requiring them to predict the same placebo mean at the primary assessment time. Compare link functions after matching response probabilities at a clinically meaningful reference point. These anchors help distinguish genuine structural consequences from arbitrary differences in parameter scale.
Before running the simulation, specify the outputs that matter: for example, mean treatment effect at the primary time, responder probability, bias, interval coverage, probability of satisfying the trial decision rule, or probability of a false positive under the null. The estimand and decision rule should remain the same across scenarios. If they change, a difference in conclusions can no longer be attributed solely to the structural choice.
Specify the feature being varied, such as link function, time course, placebo trajectory, or exposure-response shape.
Describe what each form assumes and identify clinical quantities to match, such as a reference response probability or endpoint mean.
Hold the design, population, estimand, and decision rule fixed; vary only the specified structure and any parameters that must be recalibrated.
Report scenario-specific outputs and their differences, with Monte Carlo uncertainty considered when interpreting small differences.
Link Functions: Match Clinical Effects, Not Link-Scale Numbers
For a binary response, a generalized linear model represents the response probability \(p\) through a link function \(g\): \(g(p)=\eta\), where \(\eta\) is a linear predictor. The logistic link is \(g(p)=\log\{p/(1-p)\}\); its inverse is the logistic function. The probit link is \(g(p)=\Phi^{-1}(p)\), where \(\Phi\) is the standard normal cumulative distribution function. Both map probabilities between zero and one to an unbounded predictor, but equal numerical changes on their scales do not generally produce equal probability changes.
A sound comparison therefore calibrates the alternatives to a common probability effect at a reference covariate or exposure, then examines how their predictions differ elsewhere. The following hypothetical example illustrates why this matters.
Worked Example: Calibrating Logistic and Probit Links
Suppose the placebo response probability at a reference patient profile is 0.30, and the active-treatment probability is to be 0.50 at that profile under either link. For a logistic model, the required treatment shift on the log-odds scale is
For a probit model, the corresponding shift is
Now consider a second hypothetical profile with placebo probability 0.05. Under the calibrated logistic model, its active-treatment probability is
The risk difference at this profile is approximately \(0.1094-0.0500=0.0594\). Under the calibrated probit model, the active probability is
giving a risk difference of approximately \(0.1312-0.0500=0.0812\). Both models match the specified probabilities at the reference profile, yet they imply different effects at a low-baseline-response profile. This is a structural difference in how the same reference effect extends across the probability range, not evidence that either link is correct.
The key assumption is that each link is a plausible representation of how the response probability changes with the predictor. The comparison also assumes that the reference calibration is clinically meaningful and that other aspects of the simulated trial are held comparable. If the two links are instead assigned the same numerical coefficient, their clinical effects may differ even at the reference profile; that comparison confounds link shape with effect magnitude.
Time Course and Placebo Trajectory
Treatment time course describes how the treatment effect develops or wanes. Placebo trajectory describes how the control-group outcome changes over time, which may reflect natural history, study participation, background care, or measurement effects. These are separate structural choices. A treatment effect can be time-varying even when the placebo trajectory is flat, and placebo can change over time even when the treatment effect is constant.
Useful alternatives include immediate versus delayed onset, linear versus saturating change, and sustained versus waning effect. For placebo, plausible alternatives might include no change, a linear trend, or a saturating trajectory. These forms should be calibrated to common observations when possible. Agreement at the primary endpoint does not guarantee agreement at earlier visits, in responder rates, or in the uncertainty of an estimated treatment contrast.
Worked Example: A Placebo Trajectory Changes an Earlier Responder Contrast
Consider two hypothetical placebo mean trajectories for an outcome measured in points. The saturating trajectory is
and the linear trajectory is \(m_{P,L}(t)=4t/12\). Both predict a placebo mean of 4 points at week 12. At week 8, the saturating model predicts
whereas the linear model predicts \(m_{P,L}(8)=4(8/12)=2.667\) points. Suppose, for illustration, that at week 8 treatment adds a constant 3 points to the placebo mean, outcomes are normally distributed with standard deviation 4 points in both arms, and a responder is defined as an outcome above 6 points. These assumptions give the responder contrast \(P(Y>6\mid T)-P(Y>6\mid P)\), with no additional treatment of missing data or covariates in this calculation.
For the saturating trajectory, the placebo and treatment means are 3.255 and 6.255 points. Thus the contrast is
For the linear trajectory, the means are 2.667 and 5.667 points, giving
The predicted responder treatment contrasts differ by approximately \(0.2791-0.2645=0.01467\), or 1.467 percentage points, even though the placebo means agree at week 12 and the treatment increment is 3 points in both scenarios. This calculation is a prediction under the stated normal-outcome assumptions, not a claim about an observed study.
Exposure-Response Shape
Exposure-response structure maps drug exposure to a pharmacological or clinical effect. A linear model assumes a constant change in effect per unit exposure over the modeled range. A saturating model allows diminishing increments as exposure increases. Other shapes can represent thresholds or non-monotonic responses, but each additional form requires scientific justification and enough information to support its parameters.
Worked Example: Linear Versus Saturating Exposure-Response
Suppose exposure \(C\) is measured in mg/L and effect is measured in endpoint points. Compare a linear effect \(E_L(C)=C\) points with a saturating Emax effect \(E_S(C)=10C/(5+C)\) points. Both predict an effect of 5 points at \(C=5\) mg/L, so this is the calibration anchor. At \(C=1\) mg/L, the predictions are
At \(C=10\) mg/L, they are
The models agree at the anchor but make meaningfully different predictions away from it. A trial simulation should therefore consider how its exposure distribution places participants relative to the anchor and the region where the alternatives diverge. For this comparison, exposure generation and all other trial features are held fixed; the calculation alone does not establish that exposure causes the endpoint change.
Diagnostics, Comparisons, and Limitations
Structural stress tests are most informative when the candidate forms are plausible and their predictions can be checked against available evidence. Compare predicted trajectories with empirical distributions and observed time patterns, examine whether exposure ranges cover the region where candidate curves diverge, and check whether alternative models imply impossible probabilities or implausible endpoint values. Marked divergence outside observed data is a warning about extrapolation, not proof that one structure is wrong.
For simulated trial conclusions, report the scenario-specific estimates and their differences, not only the most favorable result. Monte Carlo error matters: a small difference between estimated operating characteristics may be simulation noise. Use the earlier material on Monte Carlo error to assess its size, and avoid presenting a numerical ranking as meaningful if the uncertainty from finite replicates is comparable to that ranking. Also examine whether the decision changes, not merely whether a decimal changes.
| Approach | What it varies | Useful interpretation | Important limitation |
|---|---|---|---|
| Parameter uncertainty analysis | Parameter values within a fixed structure | Sensitivity to uncertain estimates under that model | Cannot reveal sensitivity to the chosen model form |
| Structural stress test | Plausible model forms, with parameters calibrated as needed | Stability of predictions or decisions across assumptions | Depends on which alternatives are considered and how they are anchored |
| Single preferred structure | No structural variation | A clear reference analysis | Can conceal dependence of conclusions on unsupported assumptions |
Common warning signs include large differences in conclusions after a minor-looking structural change, alternatives that were not calibrated to comparable clinical quantities, and a result driven by regions of time or exposure with little supporting data. These signals call for interpretation and, where possible, better evidence—not automatic selection of the scenario with the smallest p-value or greatest predicted benefit.
Stress tests have limits. A finite set of alternatives cannot cover every plausible model, and agreement among alternatives does not prove that they are adequate. Some structures may be weakly informed by the available data, especially in extrapolation regions. Results should therefore be reported as conditional on the stated assumptions, alongside the evidence motivating the alternatives and any consequential differences. Structural uncertainty is not eliminated by increasing the number of simulated replicates; more replicates reduce Monte Carlo error, not uncertainty about which model form is appropriate.
Check Your Understanding
Answer each question, then compare with the solutions below.
- Why should logistic and probit alternatives be calibrated to a common clinical probability effect before their predictions are compared?
- Two placebo trajectory models both predict 4 points at week 12. Does that establish equal responder contrasts at week 8? Explain.
- In the exposure-response example, what is the Emax-model prediction at \(C=5\) mg/L, and why is that value useful?
- A structural scenario changes the estimated probability of trial success by 0.002, while Monte Carlo uncertainty is about 0.01. What is the appropriate interpretation?
Solutions. (1) Equal numerical coefficients on different link scales need not imply equal probability effects; calibration separates link shape from reference effect size. (2) No. Agreement at week 12 does not constrain earlier trajectory values, and the week 8 outcome distribution and responder threshold can yield different contrasts. (3) \(E_S(5)=10(5)/(5+5)=5\) points. It is the shared calibration anchor, so differences away from it reflect shape rather than disagreement at that reference exposure. (4) The observed difference is smaller than the stated Monte Carlo uncertainty, so it is not strong evidence of a meaningful scenario difference based on these replicates; more precision may be needed before interpreting the ranking.