Comparing Computational Cost Across Designs

Why Computational Cost Belongs in a Design Comparison

A candidate sampling plan may perform well statistically but require substantially more computation to fit, or may fail more often under the analysis pipeline. Those costs matter when a simulation study compares plans: they affect how many replicates can be completed, how much computing capacity is needed, and whether the resulting performance summaries are based on many or few successful fits. A fair comparison therefore reports computational burden alongside, not in place of, statistical performance.

The comparison is meaningful only if the computational task is defined consistently. Candidate designs should be evaluated using the same fitting procedure, comparable hardware and software settings, and the same rules for recording run times and failures. If one design is fitted with different tolerances, different initial estimates, or a different timeout, the recorded cost may reflect those changes rather than the design itself.

The success criteria need to be stated before comparing designs. For example, a usable fit might require successful convergence, finite estimates, and an estimated covariance matrix that meets specified checks. The criteria should be applied in the same way to every design. A failed fit may still consume substantial time, so runtime and failure status must be recorded together. If the analysis pipeline includes simulation, data processing, and fitting, decide whether the reported runtime covers fitting alone or the full pipeline, and use that definition consistently.

Measures of Computational Burden

Suppose \(R\) fitting attempts are planned for a design. Let \(T_r\) be the runtime of attempt \(r\), and let \(F_r=1\) if that attempt fails and \(F_r=0\) otherwise. The observed failure rate is the number of failed attempts divided by the number planned. The mean runtime per attempted fit includes both successful and failed attempts.

The total runtime across attempts describes the computational work recorded for that run. It is not the same as time per fit. Nor is it necessarily the elapsed time experienced by the analyst: when attempts run in parallel, the sum of their CPU times can exceed wall-clock time. CPU time measures processor work; wall-clock time measures elapsed duration. State which one is reported. For capacity planning on a shared computing system, both can be useful.

A useful complementary measure is the observed cost per successful fit: total recorded runtime divided by the number of successful fits. This incorporates time spent on failures. If there are no successful fits, this measure is undefined; report that result rather than assigning it an arbitrary value. A design's failure rate should still be shown separately, because a single cost-per-success number can conceal whether its burden comes from slow attempts or frequent failures.

Worked Comparisons

Accounting for Repeated Attempts

If a failed fit is rerun until it succeeds, the computational burden depends on both runtime and success probability. Consider repeated, independent attempts under the same conditions, each with a constant success probability \(s\), where \(s>0\). Let \(N\) be the number of attempts up to and including the first success. The probability that at least \(k\) attempts are needed is the probability that the first \(k-1\) attempts all fail:

For a positive integer-valued random variable, its expectation is the sum of its tail probabilities. Therefore,

If attempt runtimes are identically distributed with mean \(E[T]\), and each new attempt is independent of previous attempts, the expected total time through the first success is \(E[T]/s\). To see this, attempt \(k\) is run only if the first \(k-1\) attempts failed; that event is independent of the runtime of attempt \(k\). Thus expected total time is \(\sum_{k=1}^{\infty}(1-s)^{k-1}E[T]=E[T]/s\). This calculation assumes the same success probability and runtime distribution for each attempt. It is a planning approximation, not a reason to discard observed failures or to assume that retries are free.

Putting Computational and Statistical Performance Together

Computational measures complement the statistical estimands established for a simulation question; they do not replace them. For example, a design may have an attractive interval width among successful fits but a substantial failure rate. Reporting interval performance only for successful fits describes that subset, while the failure rate describes how often the analysis did not produce the required result. Present both so readers can judge the scope of the statistical summaries.

Diagnostics, Comparisons, and Limitations

Before interpreting a cost comparison, check for conditions that can make it misleading. Confirm that runtime units and timing boundaries match; record whether attempts were run serially or in parallel; and verify that failures and timeouts have not disappeared from the runtime data. A timeout is a failed attempt under the stated criteria. Its recorded duration is the time consumed before termination, not necessarily the time the fit would have taken to finish. If timeouts are used, state the limit and report how many occurred.

Inspect the distribution of runtimes rather than relying on a mean alone. A few unusually long fits can dominate total CPU time, while a median can hide the resource burden of those fits. Compare the number of completed attempts with the planned count, and verify that success and failure classifications use the same checks across designs. If one design has no successful fits, do not compare its statistical performance as though it were simply missing at random; report the failure burden plainly.

The main alternatives answer different operational questions. Mean time per attempt is useful for estimating average work, including failed attempts. Median runtime describes a typical attempt and is less affected by a few long runs. Total CPU time describes aggregate processor demand for the run. Wall-clock time describes elapsed duration under the particular scheduling and parallelization arrangement. Cost per successful fit combines runtime and success proportion, but can obscure whether cost is driven by slow fits or failures. A concise comparison table can show several of these measures together.

A common mistake is to calculate average runtime only among successful fits and conclude that the design is computationally efficient. That excludes the time consumed by failed attempts and may make a design with frequent costly failures appear artificially fast. The correction is to report runtime across all attempts, the failure rate, and, where useful, total runtime per successful fit. Another mistake is to compare elapsed times from runs with different numbers of parallel workers as though they measured the same resource use. Report CPU time and wall-clock time with their execution conditions rather than treating them as interchangeable.

No single computational measure is universally sufficient. Runtime depends on hardware, software, workload, tolerances, and implementation; failure rates can be uncertain when the number of attempts is modest. These measures describe the tested simulation and computing setup, not an invariant property of a sampling design. Their value is in making resource demands and analysis reliability visible under clearly stated conditions.