Why Replicate Validity Matters
A simulation estimates a design’s operating characteristics by repeating a trial procedure under specified conditions. Each replicate should produce a result that follows the intended data-generating process, analysis, and decision rules. If some replicates instead contain failed model fits, non-finite statistics, or contradictory sample counts, the reported operating characteristic may depend on how those records are handled.
The key is not to label every unusual result a failure. A trial can legitimately stop an arm early, produce an extreme estimate, or yield no evidence against a null hypothesis. Those are possible outcomes of the design. A software failure is different: it means the intended computation did not produce a usable result under the prespecified procedure. A statistic can also be undefined for a mathematically legitimate data pattern; that situation requires its own classification rather than being silently repaired or discarded.
Validity is defined relative to a specific estimand and analysis. For example, a replicate may have a valid record of the trial decisions but an unusable estimate from a secondary model. Before running a simulation, specify which outputs are required for each operating characteristic and what happens if one is unavailable. This prevents a post hoc rule from deciding which results count.
Failure Categories and Their Evidence
A useful diagnostic record distinguishes at least three classes. First, a model-fit failure occurs when a required fitting procedure does not satisfy its prespecified convergence or calculation criteria. Second, a non-finite result is a required statistic or decision input that is NaN, positive infinity, or negative infinity. Third, an inconsistent sample count occurs when counts recorded at different stages of the simulation or analysis do not obey a known accounting identity. These categories can overlap, so record the underlying symptoms and a primary classification rather than assuming each replicate has exactly one problem.
A warning alone does not establish that a fit failed. Check the fitting method’s convergence status, whether the required estimates and uncertainty measures are finite, and whether the fit satisfies criteria specified before the run. Conversely, a returned estimate does not prove success if the optimizer reports non-convergence or the result violates required constraints. Save diagnostic messages and enough replicate identifiers and data summaries to reproduce the issue.
Sample-count checks must reflect the design. For each arm and stage, compare generated, randomized, eligible, and analyzed counts only where the simulation rules imply a specific relationship. A stopping rule can make the absence of later-stage observations correct. A count discrepancy is evidence of a problem only after accounting for intended stopping and the prespecified analysis population, including any outcome-missingness rules. Missing outcomes are not automatically software failures; they should be handled according to the simulation’s missing-data specification.
For each operating characteristic, state which replicate-level data, model results, and decisions must be available.
Verify counts, allocation totals, stage transitions, and arm status against the prespecified randomization and stopping rules.
Inspect convergence indicators, warnings, required estimates and standard errors, and whether required statistics are finite.
Record the replicate identifier, failure category, diagnostic message, and whether a prespecified recovery procedure was applied.
Give the planned replicate count, the number valid for each outcome, the number invalid, and the rule used to calculate each estimate.
Worked Examples
Worked Example: Estimating an Error Rate With Failed Fits
A hypothetical simulation plans \(N=1000\) replicates to estimate a type I error rate. The prespecified test is available for 993 replicates. In the other 7, a required model fit fails its convergence check, so the replicate-level test decision is unavailable. Among the 993 valid replicates, 124 reject \(H_0\) at the specified significance level.
The estimate based on valid replicates is
The valid-replicate rate is
so 99.3% of planned replicates yielded a usable test decision. The estimated rejection rate among valid replicates is about 12.49%. It estimates the intended type I error rate only if the valid replicates represent the target simulation adequately—for example, if validity is independent of the replicate’s rejection outcome—or if another justified method addresses the failures.
Dividing 124 by 1000 gives \(0.124\), but that calculation effectively treats all 7 unavailable decisions as non-rejections. It is not the estimate among valid replicates and is not justified unless the prespecified operational rule defines a failed fit as no rejection. In this example it does not. The correct report gives both the valid-replicate estimate and the failure count, and flags the potential for outcome-dependent failure.
Worked Example: A Legitimate Early Stop Versus a Count Discrepancy
In a hypothetical three-arm trial simulation, stage 1 generates 40 outcomes in each of the control, A, and B arms. The interim rule stops arm B for futility. Thus, no stage 2 outcomes are generated for B, while control and A continue. The stage 1 count is 40 for each arm, and the planned stage 2 count is 20 per continuing arm.
The realized counts are 40, 40, and 40 at stage 1 and 20, 20, and 0 at stage 2 for control, A, and B respectively. The zero for B is consistent with the futility decision: it is a valid design outcome, not a missing stage or a failed replicate. The total generated outcomes are
Suppose instead that the simulation log says 40 stage 1 outcomes were generated for A, but the required analysis record says 39 were analyzed, with no prespecified exclusion or missing-outcome explanation. The discrepancy is
That replicate should be flagged for reconciliation. The count difference does not, by itself, identify the cause: it could reflect unintended filtering, a logging error, or an analysis rule not represented in the record. Do not change the count to 40 or discard the replicate without investigating and documenting the reason.
Worked Example: Non-Finite Fit Result From a Valid Trial Pattern
Consider a hypothetical binary-outcome replicate with 12 participants per group. All 12 participants in treatment experience the event, and none of the 12 controls do. These are finite, properly recorded outcomes and their group sizes agree with the generated data. A conventional logistic regression for treatment effect may exhibit complete separation: the maximum-likelihood coefficient does not have a finite value. A software routine might consequently return an infinite coefficient or a non-finite standard error.
The trial data are not invalid merely because this outcome pattern is extreme. But if the prespecified operating characteristic requires the conventional logistic-regression statistic, the required result is not a finite, usable statistic. Record the replicate as non-estimable or failed for that analysis, according to the prespecified classification, and preserve the data and fit diagnostics. Do not replace infinity with an arbitrary large number.
A different analysis method, such as a prespecified penalized fit, may return a finite estimate. If that method was specified in advance as the analysis or fallback, apply it consistently and record that it was used. Choosing a fallback only after seeing which replicates fail changes the procedure being simulated. The distinction is therefore between a legitimate extreme trial pattern and whether the specified analysis can produce the required statistic for that pattern.
Why Complete-Case Replicate Estimates Need Care
Let \(V_i=1\) if replicate \(i\) is valid for a particular outcome and \(V_i=0\) otherwise. Let \(Y_i=1\) if the event of interest occurs in a valid replicate and \(Y_i=0\) otherwise. If \(Y_i\) is defined only when \(V_i=1\), the estimate from available replicates is
This ratio estimates the event probability among valid replicates, not automatically the event probability over all planned replicates. To see why, write \(p_V=P(V=1)\) and \(p_Y=P(Y=1\mid V=1)\). With many independent simulation replicates and a nonzero validity probability, the numerator divided by \(N\) approaches \(P(V=1,Y=1)=p_Vp_Y\), while the denominator divided by \(N\) approaches \(P(V=1)=p_V\). Their ratio therefore approaches \(p_Y\).
If failure is associated with the simulated data or trial result, \(p_Y\) can differ from the target probability. For instance, difficult-to-fit data may be more common when outcomes are unusually extreme. Reporting only the valid-case estimate can then conceal a systematic selection effect. Report the planned \(N\), the valid denominator, failure frequencies by cause, and any prespecified sensitivity analysis. Do not assume that a high valid-replicate percentage guarantees negligible bias.
Diagnostics, Alternatives, and Limitations
Useful diagnostics include cross-checking counts from the generator, randomization record, analysis input, and fitted model; scanning required outputs for NaN or infinity; reviewing optimizer convergence and warnings; and examining whether failures concentrate in particular arms, stages, scenarios, or outcome patterns. A small set of failures can still matter if it is concentrated in the very replicates most likely to meet an operating-characteristic event.
There are three broad approaches, and they answer different questions:
| Approach | What it does | Key limitation |
|---|---|---|
| Exclude invalid replicates for that outcome | Calculates the estimate using only replicates with usable results | Targets the valid-replicate subset unless validity is appropriately unrelated to the outcome |
| Apply a prespecified recovery or fallback | Uses a documented alternative computation for a defined failure condition | Must be part of the simulated procedure; may change the analysis being evaluated |
| Define an operational composite outcome | Counts a technical failure as a specified operational event, if that is the design question | Answers a different question from the statistical error rate or power alone |
A recovery method can improve completion but cannot erase the original failure. Record both the initial diagnostic and the final status. Likewise, treating every failure as a non-rejection may be appropriate only when the real trial procedure would operationally make no rejection after that failure and the estimand is explicitly defined to include that rule. It should not be introduced merely to force the denominator back to the planned replicate count.
No single numerical threshold determines whether a simulation is trustworthy. The acceptable failure rate depends on its magnitude, causes, distribution across scenarios, and possible relation to trial outcomes. If failures are frequent, clustered, unexplained, or outcome-dependent, resolve the underlying problem or qualify the operating-characteristic conclusions rather than presenting a single estimate without the diagnostics.
Common Mistakes
- Calling early stopping a failure. If an arm is correctly stopped by the interim rule, its later-stage sample count can legitimately be zero.
- Calling every warning a failed fit. Apply prespecified convergence and usability criteria; retain the warning so its relevance can be reviewed.
- Silently dropping records. State the planned and valid denominators and classify exclusions for each operating characteristic.
- Replacing a non-finite value with a convenient number. This changes the analysis and can distort results. Use only a prespecified fallback.
- Assuming the valid-case estimate is automatically unbiased. It estimates the event rate among valid replicates; outcome-related failure can make that rate differ from the target.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- In 800 planned replicates, 792 are valid for a binary operating characteristic and 96 valid replicates meet the event definition. Calculate the valid-replicate event-rate estimate and the validity rate.
- An arm is stopped at an interim analysis under the prespecified futility rule. No later-stage data are generated for that arm. Is this automatically an invalid replicate? Explain.
- A required model returns a finite coefficient but reports non-convergence. What should be checked before classifying the replicate as valid?
- Why might the estimate among valid replicates fail to estimate the operating characteristic over all planned replicates?
Solutions. (1) The valid-replicate estimate is \(96/792=0.121212\ldots\), or about 0.1212 (12.12%). The validity rate is \(792/800=0.99\), or 99.0%. These describe different quantities. (2) No. If the futility decision follows the specified rule, the absence of later data for that arm is an expected design outcome. (3) Check the prespecified convergence criteria, required uncertainty estimates and other outputs, and whether the result is usable under the analysis plan. A finite coefficient alone does not establish a successful fit. (4) Failure may depend on the simulated data or trial outcome. The valid-case estimate then describes outcomes conditional on validity, which can differ from the target operating characteristic over all planned replicates.