Randomized Ordering as a Robustness Check

Why Randomize the Update Order?

In fully conditional specification (FCS), each incomplete variable is updated from a model conditional on the other variables. A sweep consists of updating each variable once. The sequence of those updates can affect the state of the chain during a sweep: a variable updated later uses the values produced by earlier updates in that same sweep. A fixed schedule always uses the same sequence, while a randomized schedule varies the sequence.

Comparing these schedules is a sensitivity check. If important summaries from well-run fixed-order and randomized-order chains are similar relative to their Monte Carlo error, the conclusions appear less dependent on the particular sweep sequence. If they differ materially, the schedule may be interacting with the conditional models or the chain may not have explored its relevant distribution adequately. Neither outcome alone proves that the imputation models are valid or invalid.

“Randomized order” should be stated precisely in an analysis plan. One option is to draw a new permutation independently at every sweep. Another is to select one random permutation at the beginning and keep it fixed for the entire run. The first varies the order within a chain; the second is simply a fixed-order run with a randomly selected schedule. They answer related but different questions. This tutorial focuses on a new random permutation at each sweep.

For a fair comparison, keep the imputation models, predictors, parameter-draw procedure, number of chains, burn-in, and iteration budget the same. Use independent random-number streams for the runs. If only the schedule changes, observed differences are easier to interpret as schedule sensitivity rather than as consequences of other design changes.

When Is Order Theoretically Irrelevant?

There is a useful benchmark. Suppose all the conditional models are the full conditionals of one coherent joint distribution for the unknown quantities being updated, conditional on the observed data. The state should include any unknown model parameters that are updated as part of the procedure. Updating one component from its full conditional leaves that joint distribution invariant. Therefore, a fixed sequence of such updates also leaves it invariant, and a mixture over possible update sequences does too.

This statement concerns invariance, not identical finite-run behaviour. Different schedules can have different autocorrelation and transient behaviour even when they share the same invariant distribution. In ordinary FCS, the specified conditional models need not be compatible with a single joint distribution. In that case, there may be no common target distribution whose invariance explains both schedules. Comparing orders is consequently informative as a diagnostic, but it is not a test of compatibility.

The justification is short. Write \(\pi(x)\) for the joint distribution and \(K_j(x,dx')\) for an update that changes component \(j\) using its full conditional while leaving the others fixed. Full-conditional sampling gives \(\int \pi(dx)K_j(x,dx')=\pi(dx')\). Thus \(\pi\) is invariant under each \(K_j\). For a fixed order \(\pi_0\), the sweep kernel is a composition \(K_{\pi_0(1)}\cdots K_{\pi_0(p)}\); applying invariant kernels successively leaves \(\pi\) unchanged. If the order is random with probabilities \(w_{\pi_0}\), the sweep kernel is a weighted mixture of these composition kernels, so \(\sum_{\pi_0}w_{\pi_0}\pi=\pi\). The argument establishes a shared invariant distribution under the stated compatibility and full-conditional assumptions; it does not establish that an actual finite chain has reached it.

What to Compare Across Runs

Compare summaries tied to the intended use of the imputations, not only the average of all imputed cells. Useful quantities include means or proportions of imputed values by variable, summaries within clinically relevant subgroups, and estimates and standard errors from the planned substantive analysis after pooling across imputations. For each summary, retain the run-specific estimate and an appropriate MCSE. Compare the difference with its simulation uncertainty, while also considering whether the difference matters substantively in the units of the analysis.

A simple descriptive comparison for two independent runs is \(D=|\hat{\theta}_{R}-\hat{\theta}_{F}|/\sqrt{MCSE_R^2+MCSE_F^2}\), where \(F\) and \(R\) identify fixed and randomized schedules. This calculation presumes that the run summaries are independent and that their MCSEs estimate their simulation uncertainty adequately. It is a scale for judging Monte Carlo variation, not a calibrated decision rule. With dependent runs, the covariance of the estimates must also be considered; using independent streams avoids that complication in a basic comparison.

Do not substitute the standard error of the substantive estimate for the MCSE. The standard error describes sampling uncertainty under the analysis, whereas the MCSE describes numerical uncertainty from using a finite simulation to approximate a quantity. They answer different questions.

Designing and Diagnosing the Check

A randomized-order comparison is most useful when the schedule is the principal feature changed. Run multiple independent chains under each schedule when feasible, use comparable initialization strategies, and apply the same burn-in and retained-iteration rules. Record the random seed and, for randomized runs, the realized order at each sweep. The random schedule is part of the algorithm and should be reproducible.

A practical comparison can proceed in stages:

Inspect trace plots and chain summaries alongside numerical contrasts. If a randomized run has a larger MCSE, a trace with persistent trends, or summaries that depend strongly on the initial state, the comparison is not yet a clean assessment of schedule robustness. A changing schedule can itself alter the chain's short-term behaviour. Agreement between estimated means alone can conceal disagreement in tails, subgroup patterns, or other quantities relevant to the analysis.

How to Interpret Agreement and Disagreement

Substantial agreement across fixed and randomized schedules is reassuring in a limited sense: the chosen summaries do not appear highly sensitive to these schedule changes at the precision of the runs. It does not establish compatibility of the conditional models, correct specification, convergence, or validity of the missing-data assumptions. Different defective models can produce similar outputs, and both runs can share the same failure.

A difference large relative to MCSE is a reason to investigate, not an automatic reason to prefer one schedule. Check whether the difference is practically important, whether more iterations reduce the MCSE, and whether independent chains give consistent evidence. If the discrepancy persists after simulation error is reduced, report the schedule dependence and examine the conditional models and implementation. Randomizing order is not a guaranteed remedy: it can reveal sensitivity, but it cannot repair incompatible or misspecified conditional models.

Common Mistakes and Limitations

• Treating randomized order as a cure. Random order changes the update schedule; it does not make incompatible conditional models compatible or correct a misspecified model.
• Calling a difference a formal significance test. A difference-to-MCSE ratio is descriptive unless a formal testing framework and its assumptions have been specified. It has no universal threshold for declaring robustness.
• Comparing only point estimates. Differences must be interpreted alongside MCSE, relevant units, and the substantive impact of the change.
• Changing several settings at once. If models, initialization, chain length, and order all change, a discrepancy cannot be attributed to ordering alone.
• Assuming agreement proves convergence. Two runs can agree because both have not explored important regions, or because the compared summary is insensitive to a problem elsewhere.

The principal alternative is to compare several fixed permutations, rather than randomizing anew each sweep. This can identify whether one particular fixed schedule is unusual, but it requires separate runs for each selected order. A per-sweep randomized schedule samples among orders within a chain, but adds schedule variation and must be documented. Neither strategy replaces chain diagnostics, independent runs, or checks of the conditional models.