Why Failed Replicates Are Part of the Result
A simulation summary is only as trustworthy as the rules used to produce it. If some virtual patients fail during profile generation, or some exposure metrics are nonfinite, it can be tempting to remove those rows and summarize what remains. But the surviving replicates may not represent the simulation population that was planned. The failure rate, the reasons for failure, and the handling rule therefore belong in the analysis—not just in a log file.
A replicate is one planned draw or simulation unit for which the workflow is intended to produce an output. Depending on the design, that unit might be a virtual patient, a scenario-level simulation, or one draw from a parameter distribution. State the unit before counting failures: otherwise a percentage such as “2% failed” has no clear denominator.
Examples help distinguish the categories. A solver that stops before completing a concentration-time profile is a computational failure. A computed AUC of positive infinity is nonfinite. A finite, negative AUC is invalid for the usual definition of area under a nonnegative concentration profile, even though it can be averaged numerically. These labels describe what happened; they do not by themselves establish why it happened or whether the intended model is appropriate.
Define the Target and the Counting Rule
Let \(N\) be the number of planned replicates, and let \(F_i=1\) if replicate \(i\) ends in the prespecified failure state and \(F_i=0\) otherwise. The observed failure count is \(K=\sum_{i=1}^{N}F_i\), and the observed failure fraction is \(K/N\). The denominator is the number of planned independent simulation units, not the number of successful outputs and not the total number of execution attempts when retries are made.
If the planned replicates are independent draws under the same simulation protocol, each with failure probability \(q\), then \(K\) has a binomial distribution with parameters \(N\) and \(q\). In that setting, \(\hat{q}\) estimates the probability that a planned replicate fails under that protocol. Its approximate standard error is \(\sqrt{\hat{q}(1-\hat{q})/N}\), when the binomial approximation is suitable and the sample is large enough for the approximation to be informative. With different scenarios, parameter strata, or failure mechanisms, report counts and rates by group as well as any pooled rate: a pooled number can conceal where failures concentrate.
A crucial distinction is whether a failed output is merely unavailable because of a computational problem, or whether the target metric is not defined for that simulated case. If a numerical solver fails for a case where the model implies a well-defined profile, the desired output still exists conceptually, although it has not been computed. If the model or metric is genuinely undefined for a portion of the specified simulation distribution, there may be no single unconditional mean of that metric without an additional definition. Do not treat those two situations as interchangeable.
Worked Example: Classifying Failures and Calculating Their Frequency
A hypothetical workflow plans 500 virtual-patient replicates. It records 8 solver terminations, 4 completed profiles with nonfinite AUC, and 3 finite AUC values below zero. Assume these groups are mutually exclusive under the workflow’s primary-status rule.
The 8 solver terminations are computational failures. The 4 nonfinite AUC values are nonfinite outputs. The 3 negative AUC values are finite but invalid metrics. Thus \(K=8+4+3=15\) planned replicates have an unusable required AUC output.
The failure fraction is calculated using all planned replicates:
The category-specific rates are \(8/500=1.60\%\) for solver termination, \(4/500=0.800\%\) for nonfinite AUC, and \(3/500=0.600\%\) for invalid AUC. They sum to 3.00% because the stated primary-status categories are mutually exclusive. The successful-output fraction is \(485/500=97.0\%\); dividing 15 by 485 would instead compare failures with successes and would not estimate the failure probability per planned replicate.
Why Excluding Failures Can Bias a Summary
Suppose \(Y\) is the metric the simulation is intended to summarize and is conceptually defined for every planned replicate. The mean among successful replicates is \(E[Y\mid F=0]\), whereas the intended unconditional mean is \(E[Y]\). By conditioning on success, the analysis changes which simulated cases contribute. The two means agree if failure is independent of \(Y\), or in some other special cases where the conditional means happen to agree; there is no general reason they must agree.
The difference follows by partitioning the expectation into failed and successful cases:
This identity is a justification, not a correction formula for unknown missing values: if failed replicates have no observed \(Y\), then \(E[Y\mid F=1]\) is generally unavailable. It shows why the successful-only mean can differ from the target even when the failure fraction is small. Failures associated with extreme parameter draws, unusual covariates, or profiles near a metric’s problematic region are especially likely to make the missingness informative.
Worked Example: How Success-Only Averaging Changes the Target
Consider a hypothetical simulation with 200 equally weighted cases: 100 cases in a lower-exposure stratum with metric \(Y=10\) mg·h/L and 100 in a higher-exposure stratum with metric \(Y=30\) mg·h/L. Suppose numerical failures occur in none of the lower-exposure cases and in 50 of the higher-exposure cases. For illustration, assume the model’s intended metric remains defined for all 200 cases; the high-stratum failures are missing computations, not cases where the metric is undefined.
The intended mean across the planned cases is:
After silently excluding failures, 150 values remain: 100 values of 10 mg·h/L and 50 values of 30 mg·h/L. Their mean is:
The success-only result is 3.33 mg·h/L lower than the planned-case mean because failures were concentrated in the higher-value stratum. It describes the successful subset, not the original simulated population. In an actual run, the missing values are not known merely because this illustration specifies them; the example demonstrates the selection mechanism rather than offering a way to recover uncomputed outputs.
Reruns Without Changing the Simulation
A rerun is an additional execution attempt, not automatically a new planned replicate. Keep those concepts separate. A transient infrastructure interruption may be retried with the same replicate inputs and random-number assignment. If the retry succeeds, the replicate has a usable final output, but the initial operational failure and retry should still be counted and reported. By contrast, drawing a new random replicate to replace a numerical failure changes which simulated cases enter the analysis if the failed case is simply discarded.
A prespecified retry policy should answer at least four questions: which failure types are eligible for retry; whether the retry uses the same inputs and random stream; how many attempts are allowed; and how unresolved cases are reported. A change to solver settings, tolerances, software version, or model implementation is not merely “the same retry”: it changes the computational protocol. If such a change is permitted, define it in advance, record the original outcome, and assess whether the changed procedure produces outputs on the same intended target.
Never keep retrying until a replicate succeeds without a stated limit. If success probability depends on the simulated case, retaining only cases that eventually succeed creates a selected set. A fixed, documented attempt policy makes the final status interpretable and supports comparison across scenarios.
Worked Example: Counting a Retry Correctly
A hypothetical run plans 1,000 replicates. Twenty encounter a compute-node timeout. The workflow retries each timeout once using the same replicate inputs and random-number assignment. Fifteen retries complete; five time out again.
There are 20 timeout-affected planned replicates, so the initial timeout rate is \(20/1000=2.00\%\). Fifteen of those 20 are recovered operationally, and five remain unresolved. The unresolved planned-replicate rate is \(5/1000=0.500\%\); the recovered rate is \(15/1000=1.50\%\). The total number of execution attempts is 1,020, but that number is not the denominator for the planned-replicate failure rate.
A transparent report would state the 1,000 planned replicates, the 20 initial timeouts, the single same-input retry rule, the 15 recovered cases, and the 5 unresolved cases. It would not describe the run simply as having “five failures” without noting the initial failures and recovery policy. If instead each timeout were replaced by a fresh random draw and only the new draw retained, the final sample would no longer be the same set of 1,000 planned replicates.
Diagnostics, Reporting, and Limits
Failure handling should make it possible to distinguish a rare infrastructure interruption from a systematic simulation problem. At minimum, examine failure counts and rates by scenario and primary failure type, along with whether failures cluster in particular parameter or covariate regions. Inspect representative error messages and the values available immediately before failure. A rising failure rate across related scenarios, repeated nonfinite outputs, or invalid metric values concentrated in one region calls for investigation rather than automatic exclusion. These are diagnostic clues, not proof of a particular cause.
Report the planned replicate count, the number with usable outputs, the number unresolved after the stated retry policy, category-specific counts and rates, and the denominator for each rate. Explain whether summaries use all planned replicates or only successful outputs. If an output metric is undefined in some cases, state that limitation explicitly and define what population or conditional summary is being reported. A rate alone does not make a biased success-only estimate representative.
| Handling approach | What it can do | Main limitation |
|---|---|---|
| Stop and investigate | Preserves the planned analysis until the cause is understood | Delays results; does not itself resolve the failure |
| Retry the same replicate under a prespecified rule | Can recover transient execution failures without substituting a new draw | Persistent or systematic failures remain; retry outcomes still need reporting |
| Exclude unusable outputs and summarize successes | Produces a summary of the observed successful subset | May change the target and bias estimates when failure relates to the metric |
| Redesign or revise the computational method | May address a systematic numerical or implementation problem | Changes the workflow and requires transparent evaluation under the revised protocol |
There is no universal rule that converts every failed replicate into a valid value. If the metric exists but could not be computed, transparent reporting and a justified computational remedy may be possible. If the metric is genuinely undefined for some draws, a different estimand, an explicit domain restriction, or a separate description of those cases may be needed. The choice should follow the scientific question, not the convenience of obtaining a complete table.
Common Mistake
Wrong reasoning: “Only 1% failed, so dropping them cannot matter; the remaining mean is the simulation mean.” A small failure fraction does not establish that failures are unrelated to the outcome, and the remaining mean is mathematically a mean conditional on success.
Correction: State the planned target, report the failure count and rate, and investigate how failures relate to scenario and available simulation characteristics. If the metric is defined for failed cases but uncomputed, do not claim that a success-only summary estimates the unconditional target unless the required assumptions are justified. If those assumptions cannot be checked, label the summary as conditional on successful computation and discuss the limitation.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A workflow plans 800 replicates and has 12 unresolved failures after its retry policy. What is the unresolved failure rate, and what is its denominator?
- Why can the mean among successful replicates differ from the intended mean across planned replicates?
- A retry succeeds after a node interruption using the same inputs and random-number assignment. Is it a new planned replicate?
- Of 400 planned replicates, 6 have solver failures, 2 have nonfinite outputs, and 4 have finite but invalid metrics. If the categories are mutually exclusive, calculate the overall unusable-output rate.
Solutions. (1) \(12/800=0.0150\), or 1.50%; the denominator is all 800 planned replicates, not the number of successful outputs or execution attempts. (2) Success-only averaging estimates \(E[Y\mid F=0]\), which need not equal \(E[Y]\) when failure is associated with the metric or its predictors. (3) No. It is another attempt for the same planned replicate; report the recovery and the initial interruption. (4) There are \(6+2+4=12\) unusable outputs, so the rate is \(12/400=0.0300\), or 3.00%.