Sensitivity To Outcome Model Misspecification

Why Outcome-Model Sensitivity Matters

A comparison of response-adaptive randomization (RAR) with fixed randomization is only as informative as the outcome models used to describe and evaluate the designs. If a simulation assumes outcomes are less variable than they really are, an adaptive rule may appear to recognize the better treatment quickly and allocate many later patients to it. If the actual outcomes are much noisier, early apparent differences are less reliable. The resulting allocation, estimated treatment difference, power, and patient benefit can all differ from the simulation’s predictions.

This is distinct from sensitivity to the assumed treatment effect. Here, the central question is whether the assumed distribution of outcomes—including its variance and shape—is adequate. Misspecification does not have one universal consequence: a variance error directly affects uncertainty calculations, while the impact of an incorrect distributional shape depends on the rule and the statistic being used.

For the calculations below, outcomes are measured on a continuous scale, treatment assignment is randomized, and patients’ potential outcomes are independent across patients. Unless otherwise stated, the examples assume a common treatment effect: treatment B has mean outcome \(\mu_B\), treatment A has mean outcome \(\mu_A\), and \(\delta=\mu_B-\mu_A\). A positive \(\delta\) means a higher mean on the stated outcome scale; whether that is beneficial depends on the outcome’s direction. Numerical examples are hypothetical, not reports of actual trials.

What Variance Misspecification Does to Power

Consider a comparison of two sample means with fixed arm sizes \(n_A\) and \(n_B\). If outcomes in each arm are independent and have variances \(\sigma_A^2\) and \(\sigma_B^2\), then the variance of the difference in sample means is \(\sigma_A^2/n_A+\sigma_B^2/n_B\). Under a normal model, or as a suitable large-sample approximation, this gives the standard error. A power calculation that substitutes an incorrect variance uses the wrong standard error and therefore the wrong separation between the alternative distribution and the rejection threshold.

This comparison assumes the mean difference is 5 units under the alternative and that the variance is the same in both arms. It isolates variance misspecification: the effect size has not changed. In a RAR trial, the arm sizes may themselves be random and depend on earlier outcomes. Then a fixed-arm-size power calculation may be inadequate even if its variance inputs are correct; the allocation process must also be represented in the design evaluation.

How Misspecification Can Change Adaptive Allocation

Many adaptive rules use observed outcomes to change the probability of assigning future patients to each treatment. Early estimates are noisy, so the chance that an observed difference points in a particular direction depends on the outcome distribution and its variance. An assumed model that understates variability can make an early difference seem more decisive than it should be. This can change allocation and hence patient benefit, even before considering the final statistical test.

The example assumes the pilot outcomes follow the stated normal distributions, later assignments follow the stated probabilities, and the treatment effect is a constant 1 unit. It illustrates why the model matters to patient-benefit projections: under the assumed variability, the pilot appears more informative, and the allocation rule directs more patients to B on average. It does not show that every adaptive rule has this form or that higher allocation to B is always ethically preferable. The outcome scale, treatment effect, uncertainty, and trial-level objectives all matter.

Distribution Shape, Bias, and the Target of the Comparison

Getting the variance right does not ensure that the distributional model is adequate. A normal distribution and a heavy-tailed distribution can have the same mean and variance but different probabilities of extreme outcomes. Those differences can affect procedures based on thresholds, interim stopping, rankings, or transformations. In an adaptive trial, an extreme early value may influence later assignments, so tail behavior can matter to allocation as well as to standard errors.

It is important not to conclude that misspecification automatically creates bias in every estimator. With fixed arm sizes and assignment independent of outcomes, the difference in sample means has expectation \(\mu_B-\mu_A\), whether or not the outcomes are normally distributed, provided the arm means exist and the sample means are formed from the assigned patients. The normal assumption is often used for distributional calculations, not for that expectation identity. Under outcome-dependent allocation, however, sample sizes and later assignments depend on earlier responses. The joint sampling process changes, and a simple difference in sample means need not have the same properties as under fixed randomization. The size and direction of any bias depend on the rule, estimator, and outcome-generating model.

A simulation’s estimated bias is also model-dependent. If the simulation generates outcomes from an assumed distribution, its average estimate describes repeated trials under that specified distribution and design. It does not establish the same bias under a different distribution. The appropriate response is to evaluate the design under plausible alternatives, not to label the estimator biased simply because the model is uncertain.

A Practical Sensitivity Analysis

A useful comparison evaluates each design under several outcome-generating models while holding the trial question and treatment-effect scenario fixed. The assumed model should be included as one case, not treated as the only plausible case. For example, analysts can vary the standard deviations, use heavier-tailed or skewed outcome distributions when scientifically plausible, and compare results under distributions with similar means and variances. They should then report how the design comparison changes across those cases.

For every scenario, evaluate the same prespecified quantities: bias of the estimator for the target treatment contrast, variability or mean squared error, power at a stated \(\alpha\), and a clearly defined patient-benefit measure. If the benefit measure is the average outcome, its direction and units must be stated. If the clinical objective instead concerns a threshold or a patient-level utility, the relevant distributional features may differ from those that determine the mean.

Useful diagnostics include checking whether simulated outcomes reproduce plausible standard deviations, tail frequencies, and rates of extreme observations; examining how often early allocations become strongly imbalanced; and checking whether power or benefit is driven by a small number of extreme outcomes. In real data, residual plots and distributional summaries can reveal departures from a working model, but they cannot prove that a model will describe outcomes under every treatment or allocation history. Diagnostics should therefore inform sensitivity scenarios rather than replace them.

Alternatives and Limitations

A fixed randomization comparison is a useful reference because its allocation probabilities do not respond to observed outcomes. It does not eliminate outcome-model uncertainty: a power calculation can still use the wrong variance, and a patient-benefit estimate still depends on the assumed treatment effects and outcome scale. Conversely, RAR may offer an allocation advantage under some scenarios, but its benefit can be smaller or less predictable when early outcomes are noisier than assumed.

An alternative to a single parametric model is to evaluate a range of models, including distributional scenarios that preserve means and variances while changing tail behavior. Robust analysis procedures may reduce sensitivity of inference to some departures, but they do not make the allocation process independent of outcomes or guarantee robust patient benefit. No sensitivity analysis covers every possible misspecification. Its value depends on choosing scenarios that are scientifically credible and on reporting which conclusions remain stable.

Common Mistake

Incorrect reasoning: “The assumed treatment means are correct, so the power and patient-benefit comparison is reliable.” Correct means alone do not determine the standard error, tail probabilities, or how strongly early observations influence later assignment. The correction is to state and vary the variance and distributional assumptions, then recompute design-level quantities under each plausible model. Also distinguish the upper-tail contribution to power from total two-sided power: for a two-sided test, both rejection tails must be counted, even when one is small.