Monitoring Computational Cost in MICE

Why Monitor Computational Cost?

A MICE run can be statistically appropriate and still be impractical to execute, store, or repeat. A configuration that completes quickly on a small test data set may exceed a job’s time limit, compete for memory with other processes, or create output too large to retain and transfer. Monitoring computational cost makes these constraints visible before a full analysis depends on a run.

The objective is not to find one universal runtime formula. Cost depends on data dimensions, predictor structure, imputation methods, the number of imputations and iterations, software implementation, hardware, and concurrent workload. Instead, measure the actual workflow under controlled conditions, compare relevant configurations, and treat any extrapolation as an estimate to be checked.

Designing a Useful Benchmark

A benchmark is informative only if the runs being compared differ in the intended configuration rather than in uncontrolled conditions. Use the same input data, preprocessing, code path, output-retention rules, computing environment, and measurement method wherever possible. Record the configuration being tested, including the number of imputations \(m\), iteration count, number of rows and variables, and any parallel-processing settings. Record enough run identity and environment information to reproduce the comparison; a measured number without its conditions is difficult to interpret.

Measure elapsed wall-clock time when the practical question is how long a job takes to finish. CPU time is useful for understanding processor use but can exceed elapsed time for parallel work. For memory, record a process-level peak or a clearly defined alternative, such as the maximum resident set size. A final memory reading is not a peak: a run may allocate much more memory temporarily and release it before completion. Measure the saved output on disk after the same serialization and compression steps for every configuration.

Repeat each benchmark when feasible. Timing varies because of operating-system scheduling, competing jobs, cache state, and other transient conditions. Report the individual observations and a summary such as the median, rather than presenting one unusually fast run as the expected runtime. If repeated timings vary substantially, investigate the environment or increase the number of repetitions before using them to set a tight limit.

A useful benchmark also reflects the workflow that will actually be run. A test that omits saving results cannot estimate the time or storage required for saving. Conversely, including unrelated setup or data transfer in only some configurations makes their runtimes incomparable. Decide what the measurement includes, then apply that definition consistently.

Interpreting Memory and Output Measurements

Memory use is especially important because a run can fail abruptly when the process or host reaches a memory limit. Distinguish the memory attributed to the MICE process from total system memory, and distinguish peak use from memory remaining after the run. If several jobs run concurrently, each job’s peak alone does not describe whether the combined workload fits on the machine.

A rough raw-storage calculation can help explain why saved results grow, but it is not a substitute for measurement. If \(n\) rows each have \(p\) numeric values in each of \(m\) completed data sets, and each value is stored as an 8-byte number, the values alone require approximately \(8npm\) bytes. This omits the original data, object metadata, other stored run components, temporary copies, and compression effects. Actual memory and serialized file size can therefore differ substantially from the raw-value calculation.

Using Pilot Runs to Plan Resource Limits

A pilot run can estimate the cost of a larger configuration, but the estimate requires a scaling assumption. For example, when the data structure and methods are held constant, computational work may increase with \(n\), \(m\), and the number of iterations. This does not imply exact linear scaling: model-fitting costs, predictor counts, convergence behaviour, object creation, parallel overhead, and memory allocation can all change the relationship.

One simple approach is to fit a local linear approximation to measurements from two pilot sizes. Let \(T\) denote elapsed seconds and use \(n m I\) as a rough workload index, where \(I\) is the iteration count. With measurements \(T_1,T_2\) at indices \(W_1,W_2\), the fitted slope is \((T_2-T_1)/(W_2-W_1)\), and the intercept is \(T_1-\text{slope}\times W_1\). This is a planning device, not a general MICE complexity result. The estimate is most defensible near the measured range and when other settings are fixed.

Diagnostics, Alternatives, and Failure Modes

Treat a benchmark as suspect if repeated measurements vary greatly, if a run is interrupted or shares the machine with unrecorded workloads, or if the observed memory peak is implausibly lower than the memory required by the known data objects. Check that the measurement covers the same stages in every run. A file-size comparison is not fair if one configuration saves completed data sets while another saves a larger run object, or if compression differs.

For a practical workflow, set explicit limits before the full run: a maximum elapsed time, a memory allowance below the host or scheduler limit, and a storage allowance for outputs and temporary files. Keep headroom rather than planning to use every available byte or second. Use a smaller pilot to discover obvious failures, then a more representative pilot to check scaling. If a run approaches a limit, reduce workload only through a deliberate configuration decision and preserve the configuration and results needed to understand what was run.

Different assessment approaches answer different questions. A one-off timed run is quick but vulnerable to random variation. Repeated benchmarks with medians better describe typical performance but still may not reflect production contention. Analytic raw-size calculations are transparent and useful as lower-level checks, but omit implementation overhead. Pilot-based extrapolation reflects the actual software and data more closely, but is unreliable far outside the measured range. Profiling can identify where time or memory is spent, but can add overhead and may not be necessary for routine resource planning.

The main limitation is that cost is configuration- and environment-specific. Changes in predictor structure or imputation methods can alter computation even when \(n\), \(m\), and iteration count are unchanged. Parallel execution may reduce elapsed time while increasing total CPU use or memory pressure. Compression and saved-object contents affect output size. Therefore, report estimates as estimates, retain the measurements and conditions behind them, and re-benchmark when material parts of the workflow change.