Why Regression Tests Need More Than Exact Matching
A software update can change a MICE workflow in two very different ways. It might break a property that should remain unchanged, such as the number of completed datasets or the values in observed cells. Or it might alter numerical imputation draws slightly because of a change in random-number handling, computation order, or implementation. A useful regression test must catch the first kind of change without treating every difference of the second kind as a defect.
A regression test compares a new run with a defined reference run on a fixed test dataset and configuration. Its purpose is not to prove that one version is statistically superior. It is to identify departures from expected behavior, investigate them, and decide whether they are defects, acceptable implementation changes, or changes that require an updated reference. The choice of what to compare is therefore part of the test design.
Before comparison, hold the input data, missingness pattern, preprocessing, MICE settings, and any other relevant inputs fixed. The test fixture should be small enough to run reliably, but rich enough to exercise the methods and data types the test is intended to cover. If an input or setting changes, a difference in output cannot be attributed specifically to the software update.
Choose What Should Be Compared
Start by separating output properties into categories. Some properties are expected to be exact, while others are appropriately compared approximately. Treating every output value as an exact snapshot is often brittle for stochastic algorithms; treating every output as approximate can conceal serious defects.
| Property | Typical comparison | Reason |
|---|---|---|
| Number of imputations and dimensions | Exact | These are explicit configuration or structural requirements. |
| Observed values in completed data | Exact, where the workflow preserves them | Imputation should not silently overwrite observed entries. |
| Variable names, types, and allowed categories | Exact or rule-based | These are structural expectations for the test fixture. |
| Selected means, proportions, quantiles, or ranges of imputations | Numerical tolerance or replicate-based comparison | Stochastic draws and floating-point calculations may vary. |
| Every individual imputed draw | Exact only under tightly controlled reproducibility conditions | Random-number streams and computation order can change across implementations. |
A numerical tolerance should be chosen before inspecting the updated results. For a reference value \(r\) and a new value \(u\), an absolute tolerance \(\epsilon_a\) accepts the comparison when \(|u-r|\leq\epsilon_a\). A relative tolerance \(\epsilon_r\) scales the allowed difference to the size of the reference. A combined rule is:
The absolute component is useful near zero; the relative component allows a comparable percentage deviation for larger values. Tolerances should reflect practical importance and ordinary computational variability, not be enlarged merely to make a failing test pass.
Worked Example: Exact Structural Checks
Suppose a hypothetical test fixture has 12 records and three variables: a continuous outcome with 3 missing values, a binary exposure with 2 missing values, and an observed site label. The configured run requests \(m=5\) completed datasets. A test checks each completed dataset for 12 rows, 3 columns, unchanged observed entries, no missing values, and allowed exposure categories of 0 and 1.
The reference run has five completed datasets, each with 12 rows and 3 columns. In the updated run, the fourth dataset has 11 rows. The structural comparison is exact: \(11\ne12\), so the test fails regardless of whether the means or imputed values are close to their reference values. This is a meaningful defect signal because the fixed input and configuration require all 12 records to be retained.
Now suppose all dimensions match, but one observed outcome changes from 8.4 to 8.3. For a workflow expected to preserve observed values, the equality check fails: \(8.3\ne8.4\). In contrast, the imputed outcome values are not required to equal their reference draws exactly unless the test explicitly controls the relevant random stream and execution conditions. These checks separate firm structural contracts from potentially unstable stochastic values.
Numerical Comparisons and Monte Carlo Variation
A selected summary can be more stable and interpretable than a long vector of individual imputations. Examples include the mean of a continuous variable among imputed entries, the proportion assigned a category, or a quantile across the completed datasets. The metric must match the test’s purpose: a single mean can miss a change in spread or category composition, so a compact set of summaries is usually more informative than one number.
For a deterministic metric, compare the new value \(u\) to a reference \(r\) using the prespecified tolerance. For a stochastic metric, one run from each software version may differ because of Monte Carlo variation alone. In that setting, repeated runs can estimate the variation in the metric. Let \(Y_{vr}\) be the metric from replicate \(r\) under version \(v\), where \(v\) is either old or new. If the replicate runs are independent within and across versions, the estimated standard error of the difference in means is:
Here \(s_{\mathrm{new}}\) and \(s_{\mathrm{old}}\) are the sample standard deviations of the replicate metrics, and \(R_{\mathrm{new}}\) and \(R_{\mathrm{old}}\) are the numbers of runs. This standard error describes uncertainty in the difference of average metrics due to run-to-run variation under the test setup. It is not a universal acceptance threshold: the test should also consider the practical size of a change and the purpose of the metric.
Worked Example: Applying a Numerical Tolerance
For a hypothetical deterministic summary of imputed values, suppose the reference mean is \(r=10.00\) units and the updated mean is \(u=10.06\) units. The test specifies an absolute tolerance of \(\epsilon_a=0.02\) units and a relative tolerance of \(\epsilon_r=0.01\), or 1%.
The combined permitted difference is:
The observed difference is:
Since \(0.06\leq0.12\), this comparison passes. That means only that this metric falls within its prespecified tolerance; it does not show that every part of the updated imputation behavior is unchanged. A complementary check of spread, such as a selected quantile or range, may detect changes that the mean cannot.
Worked Example: Interpreting Replicate Variation
Suppose a hypothetical test is run independently 4 times under each version. The metric is the proportion of imputed binary exposure values assigned category 1. The old-version replicate values are 0.48, 0.50, 0.49, and 0.51; the new-version values are 0.50, 0.52, 0.51, and 0.53.
The means are \(\bar{Y}_{\mathrm{old}}=0.495\) and \(\bar{Y}_{\mathrm{new}}=0.515\), so the difference in means is \(0.020\), or 2.0 percentage points. The sample standard deviation is approximately \(0.0129\) for each version. Under the independent-replicate assumption, the estimated standard error is:
The observed 0.020 difference is about \(0.020/0.00912=2.19\) estimated standard errors. That value is a diagnostic signal to inspect, not an automatic verdict that the update is defective. The test should judge whether a 2.0 percentage-point shift matters for this fixture and whether the metric is designed to detect the behavior at issue. The replicate calculation also assumes independent runs and comparable run conditions; if these assumptions are not met, this standard error does not describe the comparison correctly.
A Practical Test Sequence
A disciplined sequence helps make failures interpretable. First verify that the fixture and configuration are the intended ones. Then check exact invariants before evaluating approximate numerical summaries. If an exact condition fails, a later summary that happens to match should not override that failure.
Use the same test data, missingness pattern, preprocessing, and imputation settings for the reference and updated runs.
Compare dimensions, number of completed datasets, variable structure, observed entries, and applicable category or range constraints exactly.
Use prespecified summaries and tolerances, or replicated runs when stochastic variability is important to the comparison.
Determine whether it reflects a defect, ordinary numerical variation, or an intentional behavior change; update the reference only after that review.
The logic behind using replicates is straightforward. If each replicate metric has variance \(\sigma_v^2\) under version \(v\), the variance of its sample mean is \(\sigma_v^2/R_v\). For independent old- and new-version runs, the variance of the difference of sample means is the sum of those variances, because the covariance is zero. Replacing each unknown variance with the corresponding sample variance gives the standard-error formula above. If runs are paired using a genuinely shared source of randomness, the difference should instead be formed within each pair and its sample variability estimated directly; this can be more precise, but only when the pairing is meaningful and maintained by the test setup.
Failure Modes, Alternatives, and Limits
Several common testing choices produce misleading results:
- Exact matching of every imputed value. This can fail after harmless changes to random-number consumption or computation order. Use exact matching only when the execution is deliberately controlled and exact values are part of the requirement; otherwise compare stable summaries or structural properties.
- Tolerances chosen after seeing the difference. This makes the test unable to provide a consistent alarm. Set tolerances in advance and document their practical interpretation in the test itself.
- One aggregate summary for everything. Equal means do not imply equal distributions, category proportions, or missing-cell behavior. Select multiple targeted metrics and exact invariants.
- Replacing a failed reference automatically. A new output is not necessarily correct merely because it came from the newer software. Investigate the reason for the departure before accepting a changed reference.
Regression tests complement, rather than replace, unit tests and invariant checks. Unit tests target specific functions or transformations; invariant checks verify properties that must hold regardless of software version; regression tests ask whether selected behavior changed relative to a reference. A formal hypothesis test is not automatically the best comparison: with few replicates it may have little ability to detect important shifts, while with many replicates it can flag a negligible difference. Practical tolerances and targeted checks remain necessary.
A small fixed fixture cannot represent every data pattern, model setting, or failure mode. A test that passes establishes only that the selected properties passed for that fixture and configuration. Stochastic run-to-run variability can also make replicate tests costly or noisy. Keep the test focused on high-value behaviors, and treat its result as one part of software validation rather than a guarantee of correctness.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A new version returns the same number of imputations but one fewer row in each completed dataset than the fixed fixture. Should a close imputed mean allow the test to pass?
- A reference summary is 0.00 and a new summary is 0.03. Why can an absolute tolerance be important in a combined absolute-relative rule?
- When is it inappropriate to demand exact equality of all imputed values?
- Two versions produce different replicate means. What assumptions are needed to use the independent-runs standard error formula?
Solutions. (1) No. The required row count is an exact structural invariant, and a close mean does not repair its failure. (2) When the reference is zero or near zero, a relative tolerance contributes little or nothing, so the absolute component sets a meaningful floor on the allowed difference. (3) Exact equality is inappropriate when stochastic draws or computation order may vary and the exact values are not themselves part of the software contract; use prespecified summaries, tolerances, or replicate comparisons instead. (4) The runs must be independent within and across versions, and the replicate metrics must be comparable measurements under the same test conditions; otherwise the variance calculation may not represent the uncertainty in the difference.