Assess Small Samples And Rare Outcomes

Why Sparse Data Need Their Own Assessment

A design can look satisfactory in an average-performance summary while producing a meaningful number of trials with too little information to support the planned analysis. This is especially important when the planned sample is small or the outcome is binary and uncommon. An adaptive design may allocate very unevenly, and a rare outcome may leave one or both arms with no observed events even when participants were randomized to both arms.

The simulation comparison should therefore track more than the average estimate or rejection rate. It should record whether an arm is empty, whether events are absent, whether the planned estimator or test can be calculated, and how often these events occur. The comparison is informative only if both designs face the same trial size, outcome-generating conditions, analysis rules, and definition of a failed or non-estimable replicate.

For a simulation comparison, specify the data-generating mechanism (DGM), the randomization rules, and the analysis procedure before summarizing results. A typical setup generates outcomes according to a stated model, assigns each participant using the design’s sequential allocation rule, and then applies the same prespecified analysis to each simulated trial. Under the null DGM, the treatment effect is set to its null value; the simulated rejection proportion estimates the procedure’s type I error for that particular design and scenario.

Empty Arms Under Fixed and Adaptive Allocation

In a fixed design that independently assigns each of \(n\) participants to either of two arms with probability \(1/2\), the probability that all participants go to one arm is \(2(1/2)^n\). The two all-in-one-arm events are mutually exclusive when \(n>0\). Under unequal fixed assignment probabilities, the corresponding probability is \(p^n+(1-p)^n\), where \(p\) is the probability of assignment to one arm.

For an adaptive design, allocation probabilities depend on the preceding trial history. The chance of an empty arm must therefore be calculated along the assignment path that produces that event. In particular, the probability that all assignments are to B is

Each factor is conditional on all preceding assignments being to B. It is not generally correct to multiply probabilities averaged over histories that include assignments to A. In a simulation, the direct and usually simplest assessment is to record the arm counts in every replicate and estimate the empty-arm frequency.

Rare Outcomes and Non-Estimable Replicates

An empty arm is an allocation failure, whereas a zero-event arm is an outcome-data pattern. A trial can have participants in both arms and still have no events in one or both arms. For a fixed arm size \(m\), if outcomes are independent Bernoulli variables with event probability \(p\), the probability of zero events in that arm is \((1-p)^m\). This calculation applies under those fixed-size, independent-outcome assumptions. Under response-adaptive allocation, the arm size may depend on earlier outcomes, so a binomial calculation conditional on the final arm size need not describe the zero-event probability. Simulate the whole sequential process instead.

Zero events can make some summaries or procedures undefined or unstable. For example, an observed risk is defined as zero when the arm has participants but no events; however, a log risk ratio involving that zero risk is not finite without an additional method. A zero cell can also cause separation or numerical problems in some fitted models. Do not silently replace such results with extreme finite values, omit those replicates, or treat a zero-event arm as an empty arm. State the estimator and its handling rule in advance.

Report both the proportion of all simulated replicates that are non-estimable and, when useful, performance among the estimable replicates. The latter is a conditional summary: it describes only the subset where the procedure returned a result. It cannot stand in for performance across all intended trials. For tests, specify what happens if the planned test cannot be computed. Counting a failed test as a non-rejection can affect the rejection rate; excluding it can also distort that rate if failure is related to the simulated data. Report the failure proportion alongside any rejection rate based on a defined subset.

Compare Error Rates With Paired Replicates

Small-sample analyses may not attain their nominal error rate, even when their assumptions appear reasonable. The relevant type I error is the probability that the prespecified test rejects \(H_0\) under the null DGM. Estimate it by the fraction of replicates with a rejection, using the intended analysis and significance level \(\alpha\). The finite-sample rate is a property of the combination of DGM, design, analysis, and sample size—not just the test name.

When the same simulated conditions are used for both designs, the comparison can use paired rejection indicators. Let \(I_{Ar}\) equal 1 if the adaptive design’s test rejects in replicate \(r\), and 0 otherwise; define \(I_{Fr}\) similarly for the fixed design. Write \(D_r=I_{Ar}-I_{Fr}\). The estimated difference in rejection probabilities is \(\bar D\). Since \(D_r\) is 1 for adaptive-only rejections, -1 for fixed-only rejections, and 0 otherwise, \(D_r^2\) is 1 exactly when the decisions disagree. Therefore \(E(D_r^2)=P(\text{discordant decisions})\), while \(\operatorname{Var}(D_r)=E(D_r^2)-[E(D_r)]^2\). Estimating these quantities by their paired sample proportions gives the MCSE below.

where \(\hat p_{10}\) is the proportion of adaptive-only rejections and \(\hat p_{01}\) is the proportion of fixed-only rejections. This paired MCSE reflects the replicate-level association between the two designs; calculating separate MCSEs and treating the design estimates as independent discards that information.

Diagnostics, Alternatives, and Limitations

A useful sparse-data assessment examines replicate-level counts and analysis status before relying on aggregate metrics. Check that the arm-size totals equal the planned sample size, that zero-event counts are based on participants actually assigned, and that all simulated replicates are accounted for. Summarize the distribution of arm sizes, not just their average: the minimum, maximum, and frequency of very small arms can reveal allocation patterns obscured by the mean. For binary outcomes, record the number of events per arm and flag any estimator or test that fails to return its prespecified output.

Several complementary approaches answer different questions. Direct simulation of the full sequential allocation and outcome process accommodates history-dependent assignment and is usually the primary method for the operating characteristics of interest. Exact enumeration can be useful when the sample and state space are sufficiently small, and can verify a simulation in a simplified case. Large-sample formulas may provide a benchmark, but should not be trusted as the sole assessment when expected counts are sparse or estimates approach boundaries. Fixed allocation is a comparator, not a cure for rare outcomes: it may avoid some allocation extremes but cannot guarantee events in every arm.

Monte Carlo uncertainty also limits what a simulation can establish. Increasing the number of replicates reduces simulation error, but it does not repair a misspecified DGM, an inappropriate analysis, or a poorly defined failure rule. Conversely, a wide MCSE means the operating characteristic is estimated imprecisely; it does not mean the design itself has high variability among patients. Interpret those two sources of variation separately.

• Common mistake: Treating every arm with no observed events as an empty arm. Correction: Verify the randomized arm count; an arm with participants and zero events is a zero-event arm.
• Common mistake: Excluding non-estimable replicates and reporting the remaining rejection rate as though it applied to all trials. Correction: Report the non-estimable proportion and the denominator for every performance summary.
• Common mistake: Using a binomial zero-event formula for a response-adaptive arm after conditioning only on its final size. Correction: If allocation depends on observed outcomes, simulate the joint sequential process or justify the conditional model explicitly.
• Common mistake: Concluding that two designs differ because their separate estimates differ. Correction: Quantify Monte Carlo uncertainty for the difference, using paired replicate information when the simulations are paired.