Stress-Testing Prevalence And Effect-Heterogeneity Assumptions

Why Stress-Test Enrichment Assumptions?

An enrichment design can be planned around a simple narrative: a biomarker identifies patients who benefit more, and the trial recruits those patients. The design’s operating characteristics, however, depend on quantities that may be uncertain before recruitment. The biomarker may be less common than expected, treatment effects may differ less sharply across subgroups, or the assay may misclassify more patients than anticipated. Those changes can alter both the treatment effect in the enrolled population and the number of eligible participants available.

Stress-testing makes these dependencies explicit. It does not declare one scenario to be the truth, and it does not replace simulation operating characteristics. Instead, it asks how the design behaves under a set of plausible assumptions, including assumptions deliberately chosen to challenge the design. The focus here is on the consequences of changing prevalence, effect heterogeneity, and assay performance—not on selecting an optimal enrichment rule.

From Prevalence and Assay Performance to the Enrolled Population

An assay-positive enrichment strategy recruits according to \(A=1\), not according to unobserved true status \(B=1\). The proportion of true biomarker-positive patients among assay-positive patients is the positive predictive value:

The denominator is the probability of a positive assay result. Its first term, \(\pi Se\), is the proportion of all patients who are truly positive and test positive. Its second term, \((1-\pi)(1-Sp)\), is the proportion who are truly negative but test positive. Thus, even an assay with fixed sensitivity and specificity can yield a different positive predictive value when prevalence changes.

Suppose treatment is randomized within the assay-positive group, the assay result is measured before randomization, and treatment effects within true biomarker strata are \(\delta_1\) and \(\delta_0\). If the treatment effect is defined as a difference in mean outcomes, the expected effect in the assay-positive group is the weighted average

This is the effect for the population actually enrolled under the assay-positive rule. It is not generally the effect in the true-positive subgroup. The equality follows because randomization makes the treatment groups comparable in expectation within \(A=1\), while the enrolled group contains a mixture of true-positive and true-negative patients.

Relating Scenario Assumptions to Power

For a transparent planning approximation, suppose the primary outcome in the assay-positive population is approximately normally distributed with a common standard deviation \(\sigma\) in the two randomized arms. With \(n\) participants per arm, a two-sided test at significance level \(\alpha\) has approximate standard error \(SE=\sigma\sqrt{2/n}\). Under a true assay-positive treatment effect \(\delta_{A=1}\), its standardized mean is \(\lambda=\delta_{A=1}/SE\). The approximate power is

where \(Z\) is standard normal, \(\Phi\) is its cumulative distribution function, and \(z_{1-\alpha/2}\) is the corresponding standard-normal quantile. This calculation assumes the stated normal approximation, the specified variance, the planned allocation, and a fixed analysis population. If outcome variance differs materially across subgroups or treatment arms, the variance model must reflect that; a scenario-level simulation can generate outcomes by true subgroup and estimate power under the resulting mixture.

Separate the Screening Constraint from the Randomized Sample Size

Prevalence has a second consequence when recruitment is limited by a fixed screening pool. Let \(M\) be the number of patients screened. The probability that a screened patient tests positive is \(p_A=\pi Se+(1-\pi)(1-Sp)\), so the expected number of assay-positive patients is \(Mp_A\). If all are eligible, consent, and are randomized, a rough planning calculation uses \(Mp_A/2\) per arm under equal allocation. In practice, consent, eligibility, and recruitment rates may need their own scenario assumptions.

This distinction matters. If the design can continue screening until it reaches a fixed randomized sample size, lower prevalence mainly increases the number of screens and calendar time, while effect dilution may still lower power. If screening is capped, lower prevalence can reduce both the enrolled sample size and the assay-positive treatment effect. Replacing a random eligible count with its expectation is only an approximation: when the expected count is small or close to a recruitment target, simulate the count variation rather than treating the expectation as guaranteed.

Building Useful Stress Scenarios

A scenario set should represent uncertainty that could materially affect the design, not merely produce a large table of arbitrary numbers. Start from a reference assumption set and vary prevalence, subgroup effects, and assay performance in ways that have a clinical or operational rationale. One-factor changes help identify which assumption drives a change; joint stress scenarios show whether several unfavorable but plausible conditions combine to threaten the design.

If an adaptive procedure can select a population using interim data, record selection outcomes as well as final power. For instance, if the intended target is the true-positive subgroup, a useful selection diagnostic is the probability that the procedure selects the intended population under each scenario. A high probability of selecting assay-positive patients is not the same as a high probability of selecting a population composed mostly of true-positive patients: the latter depends on \(q\). Selection based on noisy interim estimates can also be wrong even when the assay classification is accurate. The operating-characteristic simulation should therefore represent both classification uncertainty and sampling variation in the decision statistic.

Scenario values are stress conditions, not automatically probability-weighted forecasts. Averaging power across scenarios requires defensible weights representing uncertainty about which scenario applies. Without such weights, report the scenario-specific results and identify the assumptions on which acceptable performance depends.

Diagnostics, Alternatives, and Limitations

Several diagnostics can reveal when the design is fragile. A large change in \(q\) after a small prevalence change signals that assay-positive enrollment is sensitive to the prevalence assumption. A large gap between \(\delta_1\) and \(\delta_{A=1}\) indicates substantial dilution. A low \(p_A\) signals recruitment pressure if screening capacity is limited. For an adaptive design, a sharp decline in the probability of selecting the intended population under modest stress suggests that the selection decision may be unstable. These are warnings to investigate, not universal pass-or-fail thresholds.

The simplest alternative to joint stress-testing is a one-factor sensitivity analysis. It is easier to interpret because it isolates each input, but it can miss combined failures. A single best-estimate scenario is simpler still, but can give false reassurance when inputs are uncertain. A probability-distribution approach can summarize uncertainty across inputs, but requires credible distributions and dependence assumptions; it answers a different question from checking selected adverse scenarios. For operating characteristics of an adaptive design, simulation can accommodate recruitment variation, outcome distributions, and the actual decision rule, but its conclusions remain conditional on the inputs and implementation being simulated.

The calculations above also have limits. The positive predictive value uses a binary biomarker and fixed sensitivity and specificity. If assay performance varies by site or patient characteristics, one overall pair of values may hide important variation. The weighted-effect formula applies to a mean treatment effect on a specified scale; effects on other scales need not average in the same way. The normal power approximation relies on its stated variance and distribution assumptions. When subgroup-specific outcome distributions, unequal variances, missing outcomes, nonadherence, or complex adaptation matter, the simulation should represent those features rather than relying on the simple formula.

Common Mistake

A frequent error is to treat sensitivity and specificity as sufficient to determine the composition of an enriched group. They are conditional probabilities given true status; they are not the probability that a positive result is true. That probability also depends on prevalence. For example, with \(Se=Sp=0.90\) and \(\pi=0.05\), the positive predictive value is only \(0.045/0.14=0.3214\), not 0.90. Assuming that 90% of assay-positive patients are true positives would overstate the subgroup effect and may understate the required screening effort.