Why Measure Computational Cost Beside Accuracy?
Two integration strategies can achieve similar predictive performance while demanding very different amounts of computation. One may need substantially longer to fit, more peak memory, or more resources for tuning. These differences matter when a benchmark is interpreted, when a method must be run repeatedly, and when deployment resources are limited. But computational efficiency is not predictive quality: a faster method is not automatically more accurate, and a more accurate method is not automatically worth any increase in cost.
A useful comparison therefore keeps predictive metrics and computational measurements distinct, then interprets them together. To make that comparison meaningful, the benchmark must control the data size, hardware, software environment, and operations included in the measurement. Otherwise, an apparent cost difference may reflect different experimental conditions rather than a difference between integration strategies.
Define the Cost Before Timing
Wall-clock time is elapsed time from a declared start point to a declared finish point. It answers how long a user waits, including time spent waiting for a processor or reading and writing files. CPU time measures processor time used by a process; it can exceed wall-clock time when work runs in parallel across processors. Neither measure is universally preferable. Wall-clock time is often directly relevant to a user, while CPU time helps describe processor use. They should not be substituted for one another without explanation.
Memory also needs a precise definition. Peak resident memory, for example, describes the largest amount of physical memory occupied by a process during the timed operation. It is not the same as the size of a saved model or the memory occupied by input files. In parallel runs, clarify whether the reported peak is for one process or an aggregate across workers. If a method fails because it exceeds available memory, record that failure rather than reporting only the time of successful runs.
Set the measurement boundary in advance. A fit-only time excludes preprocessing and prediction; an end-to-end time may include preprocessing, model fitting, and prediction. Hyperparameter tuning can dominate total runtime, so a comparison that includes tuning for one method but not another is not fair. State whether data loading, model initialization, compilation, and any permitted tuning are included. Use the same scope for all strategies in the comparison.
The underlying assumptions are practical rather than assumptions of a probability model: the timing procedure must measure the intended operations; the compared runs must be sufficiently comparable; and repeated runs must reflect the variation relevant to the claim. Record processor and accelerator type, memory, software versions, thread limits, and whether a run is cold or warm. If timing is performed on a shared machine, competing workloads can make results unstable. Randomize or alternate the order of methods where feasible, and repeat runs under the same protocol.
Measure cost at a declared sample size \(n\) and feature count \(p\). To investigate scaling, hold one dimension fixed while changing the other. Increasing \(n\) and \(p\) together may describe a realistic workload, but it cannot identify which dimension accounts for a change in cost. Multi-omics benchmarks should also state how the feature count is defined, such as the total number of predictors across blocks or counts reported separately by block.
Summarizing Repeated Runtime Comparisons
A single timing can be unrepresentative, especially for short computations. For each repeated run, record the runtime of both methods under matched conditions. If method A takes \(t_{Ai}\) and method B takes \(t_{Bi}\) on repetition \(i\), define the paired runtime ratio as \(r_i=t_{Ai}/t_{Bi}\). A ratio greater than 1 means A took longer; a ratio below 1 means A took less time. Ratios describe relative time, while the original times retain the units and absolute scale.
When variation is multiplicative—for example, runs tend to be a percentage faster or slower—the geometric mean ratio is a useful summary. It is the exponential of the average log ratio. Pairing keeps each ratio tied to the conditions of its corresponding repetition. The log transformation also treats reciprocal changes symmetrically: a ratio of 2 and a ratio of \(1/2\) have equal and opposite log values.
Here \(m\) is the number of paired repetitions. \(R_{\mathrm{geo}}=1.25\), for example, means A's runtime was 1.25 times B's in geometric-mean terms, or 25% longer. This summary is not a claim that A is 25% slower on every run. Report the repeated observations or their spread as well as the summary, particularly when individual runs vary considerably.
Worked Example: Summarizing Paired Runtime Ratios
Hypothetical data. On the same hardware and fixed benchmark task, three paired repetitions give B runtimes of 10, 10, and 10 minutes, and A runtimes of 15, 11, and 15 minutes. The scope includes preprocessing, fitting, and prediction for both methods.
The paired ratios are \(15/10=1.50\), \(11/10=1.10\), and \(15/10=1.50\). Thus,
Equivalently, the mean log ratio is \(\{ \log(1.50)+\log(1.10)+\log(1.50)\}/3 \approx (0.4055+0.09531+0.4055)/3=0.3021\), and \(\exp(0.3021)\approx1.353\). In this hypothetical comparison, A's runtime is about 1.353 times B's in geometric-mean terms, corresponding to about 35.3% longer runtime. The individual ratios range from 1.10 to 1.50, so the summary should not be presented as if every run had the same relative cost.
Interpret Accuracy and Cost as Separate Outcomes
Predictive quality is assessed using a metric appropriate to the outcome and benchmark, such as an error measure or a discrimination metric. The metric, evaluation data, and aggregation across repetitions should be described. Computational measurements answer a different question: what resources did the strategy require to produce those predictions? Keep the reported quantities separate, including their units. Do not convert a time difference into an accuracy difference or describe a lower runtime as evidence of better prediction.
A useful joint interpretation asks whether one method is both more accurate and less costly, or whether there is a trade-off. For a fixed benchmark setting, method A dominates method B if A is at least as good on the chosen predictive metric, no worse on every cost measure being considered, and strictly better on at least one of those dimensions. If neither method dominates, the comparison involves a trade-off. The conclusion can then state what accuracy difference accompanies what resource difference, leaving the practical choice to the declared priorities and constraints.
Worked Example: A Trade-Off in Accuracy and Runtime
Hypothetical data. Two strategies are evaluated on the same benchmark task, with the same evaluation cases and measurement scope. Across the benchmark repetitions, their mean AUC values are 0.842 for strategy A and 0.838 for strategy B. Their median end-to-end runtimes are 48 and 24 minutes, respectively, and their peak memory measurements are 3.2 and 2.1 GB.
The AUC difference is \(0.842-0.838=0.004\), with A higher on this metric by 0.004. The runtime ratio is \(48/24=2.00\), so A took twice as long by the reported median runtimes. Its peak memory was \(3.2-2.1=1.1\) GB greater. These results describe an accuracy-cost trade-off: A has the higher reported mean AUC, while B requires less time and memory. They do not, by themselves, establish whether the AUC difference is reliably distinguishable from benchmark uncertainty, nor whether a 0.004 difference is practically important. That judgment requires the study's uncertainty assessment and a decision context.
A Pareto comparison is useful when there are several cost measures. A method that is faster but uses more memory may not be an unambiguous computational winner. If the benchmark has practical limits—such as a maximum fitting time or memory capacity—report whether each method meets them. Avoid hiding such constraints inside an arbitrary single “efficiency score” unless the score and its weights have a clear, justified interpretation.
Assessing How Cost Changes With Data Size
Scaling measurements describe how observed cost changes as \(n\) or \(p\) changes under a specified benchmark. They are empirical summaries over the tested range, not proofs of an algorithm's asymptotic complexity. For two sample sizes with feature count fixed, an approximate power relationship \(T(n)=c n^q\) implies that the exponent can be estimated from the runtime ratio:
This follows by taking logs of the assumed relationship: \(\log T(n)=\log c+q\log n\). Subtracting the equation at \(n_1\) from the equation at \(n_2\) gives \(\log\{T(n_2)/T(n_1)\}=q\log(n_2/n_1)\); dividing by the latter log ratio yields the expression. The power relationship is only an approximation, and two points provide a rough summary rather than a reliable account of behavior over all sizes. More sizes can reveal curvature, abrupt changes, or bottlenecks.
Worked Example: Estimating Sample-Size Scaling
Hypothetical data. With the feature count and hardware held fixed, a method takes 20 seconds at \(n=500\) and 91 seconds at \(n=2000\). The sample size quadruples, so the two-point scaling estimate is
Over these two measured sizes, the observed time increase is roughly consistent with a power exponent of 1.093. This does not prove that the method has complexity \(O(n^{1.093})\). The estimate may reflect fixed overhead, implementation details, or machine effects, and it should be checked at additional sample sizes. To examine feature-count scaling, the analogous comparison must hold sample size fixed while varying \(p\).
Diagnostics, Alternatives, and Limitations
Before interpreting a cost comparison, check whether the measurement actually represents the intended workload. Very short runs may be dominated by timer resolution or setup overhead; repeated runs may instead benefit from caching, compilation, or warm-up. Shared-machine activity, different thread counts, GPU synchronization, file I/O, or a different hardware accelerator can create misleading contrasts. Confirm that the timed operation starts and stops at the stated boundaries, that devices have completed asynchronous work before timing ends, and that each strategy uses the declared parallelism.
Inspect the full set of repeated measurements. A large spread, a single unusually long run, or a method that sometimes fails indicates that a lone mean or best-case time is not an adequate account of cost. Report failed runs and memory limits. When measurements are skewed or vary multiplicatively, medians and geometric summaries can be more informative than arithmetic means alone; whichever summaries are used, show the units and describe the repetitions. If cost differences are small relative to run-to-run variation, do not state a firm ranking without appropriate uncertainty assessment.
There is no single cost measure that answers every practical question. Wall-clock time supports claims about elapsed delay; CPU time describes processor use; peak memory is relevant to whether a workload fits available hardware. Energy use or monetary cost may matter in a particular setting, but only when measured under a consistent, stated procedure. Reporting several quantities is often more informative than selecting one as a universal measure of efficiency.
A fixed-budget comparison and a fixed-accuracy comparison answer different questions. Under a fixed tuning or compute budget, methods are compared on what they achieve with the same allowance. A fixed-accuracy comparison instead asks how much resource is needed to reach a specified predictive target; it requires sufficient measurements to establish whether that target was reached. Do not compare methods under different tuning allowances and then attribute the difference solely to integration strategy. Even with a careful protocol, results are conditional on the tested data dimensions, implementation, software, and hardware; another setting may have a different cost profile.
Putting the Comparison Together
Name the data dimensions, hardware, software, operations included, and cost measures with units.
Use the same benchmark conditions, resource settings, and declared computational budget for each strategy.
Record repeated times and resource use, check failures and unusual variation, and summarize the measurements transparently.
State predictive performance and cost results in their own units, then describe any trade-off or dominance in context.
The central reporting discipline is simple: state what was measured, under what conditions, and how the number should be interpreted. That makes it possible to recognize a method that is computationally attractive without claiming that it predicts better, and to recognize a predictive gain without concealing the resources required to obtain it.
Check Your Understanding
Answer the questions, then compare your reasoning with the solutions below.
- A strategy takes 18 minutes in one run and 12 minutes in another strategy's run. What is the runtime ratio, and what does it mean?
- Why is it not possible to identify sample-size scaling if both sample size and feature count change between two benchmark runs?
- Strategy C has a higher reported predictive score but also takes longer and uses more peak memory than strategy D. Does C dominate D? Explain.
- Name two conditions that should be controlled or reported when comparing runtime across methods.
Solutions. (1) The ratio is \(18/12=1.50\). The first strategy took 1.50 times as long, or 50% longer, in those runs; this says nothing by itself about predictive quality. (2) Both dimensions could contribute to the cost change, so their effects cannot be separated. Vary sample size while holding feature count fixed to examine sample-size scaling. (3) No. C is better on the stated predictive score but worse on both stated cost measures, so neither strategy dominates the other on these outcomes. (4) Examples include hardware, software versions, thread limits, warm-up status, and the operations included in the timing. A fair comparison must use or report comparable conditions.