Why Computational Performance Is Part of Method Comparison
Two multi-omics methods can produce similarly useful results while demanding very different amounts of time, memory, or computational effort. A method that frequently fails to converge may also be impractical, even if its successful fits look good. Comparing methods therefore requires more than recording a runtime beside an accuracy score: it requires specifying what was run, under what conditions, and what counts as a completed fit.
A fair comparison starts with a fixed benchmark scenario and a prespecified computing protocol. The protocol should specify the hardware and operating environment, software versions, resource limits, parallel settings, data supplied to each method, and the timing boundary. It should also define success and failure before results are examined. These choices make the reported measurements interpretable; they do not make results transferable to every computing environment.
Standardize the Measurement Before Running the Benchmark
A runtime is meaningful only in relation to its measurement boundary. Wall-clock time measures elapsed time from a specified start event to a specified end event. For example, a benchmark might start the clock when the method begins reading prepared input files and stop it when a valid result has been written. If data preprocessing, model tuning, or output conversion is excluded, say so. If the intended use is an end-to-end pipeline, those steps should instead be included consistently.
Record the machine and its relevant resources, such as processor, memory, operating system, and any accelerator. Record software and library versions, thread counts, and resource limits. Avoid running competing jobs that unpredictably consume the same resources. If methods are intended to run in parallel, define that setting consistently; otherwise, fix the number of threads. State whether a run uses a fresh process, whether compilation or initialization is timed, and how any warm-up runs are handled. These decisions can materially change measured time.
Peak memory should also have a defined measurement. A common choice is peak resident set size (RSS), the largest amount of physical memory held by the process during a run. Report its units and say whether the measurement includes child processes. If an accelerator is used, its memory may need to be recorded separately: a host-process RSS value does not describe accelerator memory use. Average memory is not a substitute for peak memory when the question is whether a method exceeds an available memory limit.
Repeated measurements help reveal variation caused by operating conditions or stochastic computation. Report the number of repetitions and the summary used, such as median and range. For method comparisons, use the same benchmark inputs and resource protocol where possible. A single timing is particularly vulnerable to temporary system load and should not be presented as a stable property of a method.
Worked Example: Summarizing Runtime and Peak Memory
Suppose a hypothetical method is run five times on the same prepared input. Each run uses a fresh process on the same machine, with one thread and no competing jobs. The timing begins at input reading and ends when a valid fit is saved; warm-up runs are excluded. The measured wall-clock times are 41, 44, 43, 46, and 46 seconds. Peak RSS values are 1.18, 1.21, 1.19, 1.25, and 1.17 GiB.
The mean runtime is
The ordered times are 41, 43, 44, 46, and 46 seconds, so the median runtime is 44 seconds and the range is 41–46 seconds. The ordered peak RSS values are 1.17, 1.18, 1.19, 1.21, and 1.25 GiB, giving a median peak RSS of 1.19 GiB. A concise report could state: “Across five successful runs under the specified one-thread protocol, median wall-clock time was 44 seconds (range 41–46), and median peak RSS was 1.19 GiB.” These measurements describe the stated setup, not an environment-independent runtime guarantee.
Define Success and Classify Failures
“Converged” should not be used as a vague synonym for “worked.” A method-specific convergence criterion might require an optimization algorithm to meet a stated tolerance, or an iterative procedure to satisfy a documented stopping rule. Completion may additionally require that the process exits normally and produces the expected outputs. Output validity can require finite parameter estimates, correct dimensions, and satisfaction of other prespecified checks. State the criteria in terms appropriate to the method; do not assume that two algorithms have identical internal convergence diagnostics.
Classify unsuccessful attempts rather than merging all of them into a single unexplained category. Useful categories include non-convergence at the iteration limit, numerical failure such as non-finite estimates, a crash, a memory-limit failure, a timeout, and an external infrastructure interruption. A timeout is not a completed runtime: the method has not been observed to finish by the limit. Record the limit and the time reached, and count the attempt as unsuccessful under the prespecified completion rule. If an infrastructure interruption warrants a rerun, define that rule in advance and document the interruption and rerun; do not rerun selectively because a method produced an inconvenient result.
The denominator for a success rate is the number of planned attempts, not only the runs that returned a result. Keep method-specific failure counts and state whether a failure occurred before or after substantial computation. For successful-fit runtime summaries, identify that the summary is conditional on success. Report timeout frequency separately rather than treating the timeout limit as if it were the completed runtime.
Worked Example: Calculating a Success Rate
In a hypothetical benchmark, one method is run on 20 planned inputs with a 120-second limit per attempt. Seventeen attempts finish, meet the stated convergence and output-validity criteria, and save a valid fit. Of the other three, one reaches the iteration limit without convergence, one produces a non-finite estimate, and one reaches the time limit without completing.
The estimated success proportion is
Thus 85% of planned attempts produced valid fits, while 3 of 20, or 15%, did not. The failures should be reported by type: one non-convergence, one numerical failure, and one timeout at 120 seconds. A runtime summary based only on the 17 successful fits must be labelled as conditional on success. The timed-out run is not evidence that the method’s completed runtime was exactly 120 seconds; it is evidence that the method did not complete within the prespecified limit.
Keep Conditional Fit Quality Separate from Pipeline Reliability
If a fit fails, a method-specific recovery or predictive score may be undefined. Omitting failures and reporting only the score among successful fits answers a conditional question: how well did the method perform when it returned a valid fit? That can be useful, but it does not describe the experience of applying the full method to every planned input. Methods with different success rates can look deceptively similar if failures disappear from the score summary.
A second, explicitly defined summary can represent end-to-end performance if the score and failure treatment support it. For example, if a loss is bounded between 0 and 1 and the benchmark’s operational goal justifies assigning loss 1 to an unsuccessful fit, the resulting mean loss includes both successful and failed attempts. This is a choice about the benchmark’s target, not a universal rule. If no defensible failure penalty exists, report the success rate and the conditional score separately instead of inventing a penalty.
This follows by conditioning on the two mutually exclusive outcomes, success and failure: the overall expectation is the success probability times the mean loss among successes, plus the failure probability times the assigned failure loss. The equation makes explicit why a conditional mean alone cannot represent performance across all attempts when failures occur.
Worked Example: Accounting for an Incomplete Fit
A hypothetical method is attempted on five inputs. Four attempts produce valid fits with losses 0.2, 0.3, 0.1, and 0.4; the fifth fails. Suppose the benchmark has a prespecified operational interpretation in which an unsuccessful fit receives the maximum possible loss, \(F=1\), on a 0-to-1 scale.
The mean loss among successful fits is
This is the conditional loss among the four completed fits. The success proportion is \(4/5=0.8\). Including the prespecified failure loss, the all-attempt mean is
The two values answer different questions: 0.25 describes loss among successful fits, while 0.40 describes the stated operational loss across all five attempts. If the benchmark did not justify assigning loss 1 to failure, only the conditional loss and success rate should be reported.
Diagnostics, Alternatives, and Limitations
Before interpreting a computational comparison, check that every method received the intended inputs and resource allocation, that timing boundaries were applied consistently, and that units and software versions are recorded. Inspect failures by type and scenario. A sudden cluster of timeouts, memory-limit failures, or numerical errors may indicate an implementation or configuration problem rather than an intrinsic property of the method. Check that repeated runs actually used the specified thread count and that one method was not inadvertently timed with extra preprocessing or initialization.
Different summaries answer different questions. Mean runtime uses all observed completion times but can be pulled upward by a few long runs. Median runtime describes a typical observed completion and is less sensitive to unusually long times, but does not show the upper tail by itself. Wall-clock time measures elapsed time; CPU time measures processor use and can be misleading as a measure of user waiting time when methods use parallel processing. Peak RSS concerns memory demand; average memory use may conceal a short, consequential peak. No single summary replaces reporting the measurement definition and repetitions.
| Measure | What it describes | Important qualification |
|---|---|---|
| Wall-clock time | Elapsed time over a stated timing interval | Depends on computing conditions and timing boundaries |
| CPU time | Processor time consumed | Can exceed elapsed time when work is parallelized |
| Peak RSS | Maximum resident host memory used | May not include accelerator memory or all child processes |
| Success rate | Proportion of planned attempts yielding valid fits | Requires explicit completion criteria and a defined denominator |
A common mistake is to calculate a method’s average runtime or score only from successful fits and then describe the result as the method’s general performance. That discards precisely the runs that reveal reliability problems. The correction is to report the number of planned attempts, classify failures, label successful-fit summaries as conditional, and—only when justified—add an explicitly defined all-attempt score. Another mistake is to compare raw times from different machines as if they were method properties. Standardized conditions support a fairer within-benchmark comparison, but measurements still depend on hardware, software, and workload.
These measurements also have limits. A method’s runtime on one input size need not predict its runtime on a larger feature set, a different missingness pattern, or a different computing platform. A success rate summarizes the tested attempts; it does not establish that every future run will succeed at that rate. Recording runtime, memory, and reliability alongside statistical performance makes the benchmark more informative, but does not replace examining whether the simulated scenarios are appropriate for the intended comparison.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A method is run on 12 planned inputs. Nine fits are valid, two fail to converge, and one times out. What is its success rate, and what should be reported about the timeout?
- Why is a median wall-clock time not enough to characterize computational performance?
- A method has a mean score of 0.18 among successful fits and a 70% success rate. Can the all-attempt score be calculated without additional information?
- Name two protocol details that should be standardized or reported when comparing peak memory use.
Solutions. (1) The success rate is \(9/12=0.75\), or 75%. Report the two non-convergences and the timeout separately, including the prespecified time limit; do not describe the timeout limit as a completed runtime. (2) The median describes a typical observed elapsed time, but does not show the range or upper tail, and excludes failures unless the reporting explicitly accounts for them. Report repetitions and an additional summary such as the range, as well as failure counts. (3) No. An all-attempt score requires a rule for handling failed fits and the relevant scores or assigned failure value. Without a defensible failure rule, report the conditional score and success rate separately. (4) Examples include the machine and available memory, whether peak RSS includes child processes, whether accelerator memory is recorded separately, and the measurement tool and units. State enough detail for readers to know what the reported peak covers.