Why Evaluate Operational Size and Duration?
A design can meet its inferential goals in simulation and still be difficult to run. It may enroll more participants than a site network can recruit, yield fewer analyzed outcomes than expected, or take longer than the available funding period. These are questions about the distribution of operational outcomes across simulated trials, not just about a single planned sample size or an average duration.
The unit of summarization is the simulated trial replicate. For a given data-generating scenario and design, each replicate produces an enrollment count, an analyzed sample size, and a duration. These quantities can vary because the design uses adaptive rules, because outcomes or follow-up are variable, or both. A useful assessment describes that variation and estimates how often specified limits are exceeded.
Enrollment and analyzed sample size are not interchangeable. Enrollment might count participants randomized, while analyzed sample size counts participants whose data contribute to the defined primary analysis. The protocol and simulation algorithm must specify both definitions. If all enrolled participants contribute to the primary analysis, the counts may be equal; if outcomes are unavailable or participants are excluded under the analysis definition, \(A_r\) may be smaller than \(E_r\). Do not assume a fixed difference or ratio without a model that supports it.
Duration also needs an operational definition. For example, it could be the elapsed calendar time from first enrollment to availability of the primary analysis data. If the simulation ends follow-up at a different point, report that endpoint instead. State the unit and endpoint so that a duration limit has an unambiguous interpretation.
Summarizing the Distributions
For enrollment and analyzed sample size, report a measure of typical size and a measure of spread. The mean describes the average across replicates; the median describes the middle replicate when the values are ordered. Quantiles and the minimum and maximum convey how far the distribution extends. For duration, which can have a long right tail when some simulated trials take much longer than usual, the median and upper quantiles can be more informative about operational planning than the mean alone.
An empirical quantile convention should be stated when it matters. One simple convention for the \(q\) quantile is the order statistic at rank \(\lceil Mq\rceil\), where the replicate values are sorted from smallest to largest. Other software may interpolate between order statistics, so small samples can produce different reported quantiles. The same convention should be used when comparing designs.
These means summarize the simulated distributions; they are not guarantees about an individual future trial. A median duration of 18 months means half the simulated durations are at or below 18 months and half are at or above it, subject to ties. It does not mean every trial will finish in 18 months. For a design with a strict operational deadline, the upper tail and the probability of crossing the deadline are particularly relevant.
Worked Example: Reporting Typical Enrollment and Duration
Suppose 10 hypothetical replicates of one design under one scenario produce enrollment counts \(80,85,90,90,95,100,100,105,110,125\). Their durations, in months from first enrollment to primary-analysis data availability, are \(14,15,16,16,17,18,18,20,22,28\). The values are already sorted.
Mean enrollment. The total is \(980\), so the mean is \(980/10=98.0\) participants. The fifth and sixth values are 95 and 100, giving a median of \((95+100)/2=97.5\) participants. Under the stated nearest-rank convention, the 90th percentile is the ninth value, \(110\) participants.
Mean duration. The durations total \(184\) months, so the mean is \(184/10=18.4\) months. The median is \((17+18)/2=17.5\) months. The nearest-rank 90th percentile is the ninth duration, \(22\) months.
The 28-month replicate pulls the mean duration above the median. Reporting only the mean would obscure that most simulated trials finished sooner, while reporting only the median would hide the slow upper tail. Both summaries, together with a high quantile, give a more useful planning picture.
Analyzed sample size should be summarized separately, using the replicate-level \(A_r\) values. It may be useful to show enrollment and analyzed sample size side by side because operational limits may apply to either one. The difference \(E_r-A_r\) can be summarized if that gap is relevant, but it should not replace either count: a small average gap can conceal replicates with substantial loss of analyzable data.
Estimating the Probability of Exceeding a Limit
A probability statement must identify the scenario, the event, and the limit. For example, “the probability that enrollment exceeds 120 participants under scenario S” concerns the proportion of trials generated under scenario S whose enrollment is greater than 120. The limit may instead be inclusive: “at least 120” includes values equal to 120. Use the inequality that matches the operational rule.
This estimate is justified because each indicator records whether one simulated trial crosses the limit. Its average is exactly the fraction of simulated trials that cross it. If the replicates are independently generated under the same specified scenario and design, the indicator outcomes represent repeated draws of that event, and the fraction estimates its probability under that simulation model.
Worked Example: Enrollment Above a Capacity Limit
In 20 hypothetical replicates, enrollment counts are \(80,90,95,100,105,110,115,118,120,120,121,122,125,128,130,135,140,145,150,160\). The stated capacity rule is that enrollment must not exceed 120 participants.
For the event \(E_r>120\), the counts 121, 122, 125, 128, 130, 135, 140, 145, 150, and 160 exceed the limit. Thus 10 of 20 replicates exceed it: $$ \widehat{P}(E>120)=\frac{10}{20}=0.50. $$
The estimated probability of exceeding the cap is 50% under this simulated scenario. If the rule instead treated enrollment of exactly 120 as a violation, the event would be \(E_r\geq120\). Two replicates equal 120, so the estimate would be \(12/20=0.60\). The distinction is substantive, not a formatting detail.
Joint Limits and Dependence
A design may have to satisfy more than one operational constraint. For instance, a sponsor might require enrollment not to exceed a recruitment capacity and duration not to exceed a funding deadline. Estimate the probability of violating both constraints together by checking both conditions within each replicate. Do not calculate a joint probability by multiplying marginal probabilities unless independence has been established; enrollment and duration can be related through the design’s stopping and accrual rules.
Worked Example: Enrollment and Duration Constraints Together
Consider eight hypothetical replicates with an enrollment cap of 120 participants and a duration limit of 18 months. The table reports enrollment, analyzed sample size, and duration. Duration is measured through primary-analysis data availability.
| Replicate | Enrollment \(E_r\) | Analyzed \(A_r\) | Duration \(T_r\) (months) |
|---|---|---|---|
| 1 | 110 | 104 | 16 |
| 2 | 115 | 108 | 17 |
| 3 | 125 | 118 | 19 |
| 4 | 130 | 121 | 20 |
| 5 | 105 | 99 | 18 |
| 6 | 122 | 114 | 22 |
| 7 | 118 | 110 | 21 |
| 8 | 140 | 130 | 24 |
Enrollment exceeds 120 in replicates 3, 4, 6, and 8. Duration exceeds 18 months in replicates 3, 4, 6, 7, and 8. Both limits are exceeded in replicates 3, 4, 6, and 8, so the joint estimated probability is \(4/8=0.50\). At least one limit is exceeded in five replicates, giving \(5/8=0.625\). Replicate 5 has duration exactly 18 months, so it does not exceed a limit defined using \(T_r>18\).
The marginal estimates are \(4/8\) for enrollment and \(5/8\) for duration. Multiplying them would give \(0.50\times0.625=0.3125\), which is not the observed joint estimate of \(0.50\). The joint event must be counted directly unless independence is justified.
Assumptions, Diagnostics, and Design Comparisons
These summaries describe the simulated design only to the extent that the simulation represents the intended design and scenario. The protocol rules must be implemented correctly, the simulated scenario must encode the assumptions of interest, and the replicate outputs must use consistent definitions. In particular, check that enrollment is counted at the same trial stage in every replicate and that duration uses the same start and stop events. If simulation runs terminate early because of a coding or computation failure, do not silently treat those runs as ordinary completed trials.
Inspect distributions, not just summary columns. A histogram or empirical distribution plot can reveal skewness, clusters, unexpectedly fixed values, or a small number of extreme durations. Check minimum and maximum values against the design’s rules: a simulated enrollment greater than a hard maximum may indicate that the algorithm or the counting definition needs investigation. Also examine whether analyzed sample size can fall below a level needed for the planned analysis, if such a threshold is operationally important.
When comparing designs, report results separately for each scenario. Pooling replicates from scenarios with different assumptions creates a mixture whose summaries depend on how many runs were assigned to each scenario; it does not describe either scenario’s operating distribution. A compact comparison can show the mean, median, upper quantile, and limit-exceedance probability for each design-scenario combination. Use the same limits, quantile convention, and outcome definitions throughout.
A fixed planned sample size can be a helpful reference, but it is not a substitute for the simulated distribution when the design can stop early, expand enrollment, or otherwise adapt. The planned number may be a target or maximum, while the simulated enrollment distribution describes what the rules actually produce. Likewise, an average duration is useful for broad comparison but may be inadequate for a deadline decision: a design with a modest mean can still have an operationally important upper tail.
Specify what counts as enrolled, what qualifies for the primary analysis, how duration is measured, and whether each limit is strict or inclusive.
Calculate replicate-level means, medians, quantiles, and ranges for enrollment, analyzed sample size, and duration.
For each replicate, evaluate the exact operational event; divide the number of occurrences by the number of simulated replicates.
Inspect distributions and possible implementation problems, then state the scenario, units, event definition, and limits alongside each result.
Common Pitfalls and Limitations
- Reporting only the mean. A mean can hide a long duration tail or a group of replicates near a capacity limit. Include distributional summaries and relevant exceedance probabilities.
- Using “sample size” without defining it. Enrollment and analyzed sample size can differ. Name the count and state its operational definition.
- Changing the inequality silently. “Exceeds 120” means greater than 120; “reaches or exceeds 120” means greater than or equal to 120. Count the event specified by the rule.
- Multiplying marginal probabilities. Separate estimates for enrollment and duration do not determine their joint probability without an independence assumption. Count the joint event in each replicate.
- Pooling distinct scenarios. A pooled distribution can mask poor performance in an important scenario. Show scenario-specific results before any justified aggregation.
Finally, an empirical exceedance fraction is an estimate based on a finite simulation, not a certainty about future operations. Its precision depends on the number of simulated replicates, especially when crossings are rare. The simulated probability is also conditional on the design implementation and scenario assumptions; it does not account automatically for operational conditions omitted from the simulation.
Check Your Understanding
Answer each question using the definitions and rules stated in the question.
- In 12 replicates, 3 trials enroll exactly 100 participants and 2 enroll more than 100. What is the estimated probability of exceeding 100, and what is the probability of enrolling at least 100?
- Why should enrollment and analyzed sample size be reported separately if some enrolled participants lack primary-outcome data?
- In 10 replicates, the mean duration is 20 months and the median is 17 months. What does the difference suggest, and which additional summary could help assess a 24-month deadline?
- Four of 10 replicates exceed an enrollment cap, and 6 of 10 exceed a duration limit. Is the joint probability of exceeding both limits necessarily 0.24? Explain.
Solutions. (1) Exceeding 100 occurs in 2 replicates, so the estimate is \(2/12\approx0.167\). At least 100 occurs in \(3+2=5\) replicates, so the estimate is \(5/12\approx0.417\). (2) The counts measure different quantities: enrollment records participants entering the trial, while analyzed sample size records those included in the primary analysis. Missing primary outcomes can make the latter smaller. (3) The mean being above the median is consistent with a right-skewed distribution or a few long trials. Report an upper quantile or the fraction with duration greater than 24 months to assess the deadline directly. (4) No. The events may be dependent; count replicates in which both limits are exceeded. The marginal fractions alone do not determine that joint fraction.