Why the Saved State Matters
In fully conditional specification (FCS), each update changes the current imputations for one variable’s missing values, conditional on the other variables’ current values. A full sweep applies the specified updates to all variables with missing values, in the chosen order. The chain therefore passes through a sequence of states, and saving data sets means selecting particular states from that sequence.
A saved data set is not a record of every intermediate calculation, nor is it an average of the values visited by the chain. It is a snapshot: the observed data together with the current imputed values at a specified point in the iteration. Defining that point precisely makes the saved data reproducible and avoids mixing values from different stages of an update.
This definition concerns the variables designated for imputation. If the analysis includes other variables that are not being imputed, their values remain as recorded; “completed” does not mean that every conceivable field in a data file must be filled in. The essential bookkeeping rule is that a saved state has one clearly identified iteration number and includes the results of every update in the sweep.
Choosing Which Iterations to Save
A retention schedule specifies how many initial sweeps to leave unsaved and how far apart to save subsequent states. Let \(B\) be the number of initial full sweeps not retained, \(L\) the interval between retained states in sweeps, and \(m\) the number of data sets to save from the chain. Under the convention that the first saved state is \(L\) sweeps after the \(B\) discarded sweeps, the retained iteration numbers are:
Thus the final saved state is at sweep \(B+mL\). If \(L=1\), save after every sweep once the initial \(B\) sweeps have been passed. If \(L>1\), the intervening states are visited by the chain but not retained as completed data sets. The interval is a storage and sampling choice; it does not, by itself, establish that retained data sets are independent or that the chain is suitable for inference.
The schedule presumes that the FCS updates can be carried out as specified: each variable’s conditional model is appropriate for its type and conditioning set, the updates generate the intended stochastic draws, and observed values remain fixed. For inferential use, the models and the chain’s behavior also need adequate assessment. A retention schedule cannot repair misspecified conditional models or establish that the chain has reached a suitable regime.
Worked Example: Saving Snapshots After Complete Sweeps
Consider a hypothetical four-person data set with variables \(X\) and \(Y\). The observed values are shown below; the symbol “missing” marks cells to be imputed.
| Person | \(X\) | \(Y\) |
|---|---|---|
| 1 | 10 | 2 |
| 2 | missing | 4 |
| 3 | 14 | missing |
| 4 | 18 | 8 |
Suppose the update order is \(X\), then \(Y\). After one complete sweep, the hypothetical conditional draws yield \(X_2=12.4\) and \(Y_3=6.1\). After a second complete sweep, they yield \(X_2=11.7\) and \(Y_3=5.6\). These values are illustrative draws, not estimates calculated from the four observed rows.
If the schedule is \(B=1\), \(L=1\), and \(m=1\), the first retained state is at \(t_1=1+(1)(1)=2\). The saved data set is therefore:
| Person | \(X\) | \(Y\) |
|---|---|---|
| 1 | 10 | 2 |
| 2 | 11.7 | 4 |
| 3 | 14 | 5.6 |
| 4 | 18 | 8 |
The first sweep is not retained, but it still contributes to the chain’s second state. The observed entries remain as recorded, while each originally missing entry has its iteration-2 imputation. The values 12.4 and 6.1 belong to the iteration-1 state, so combining one of them with an iteration-2 value would not save the specified snapshot.
Why Save After a Full Sweep?
Within a sweep, a variable is updated using the current values of the other variables. After the first variable has been updated but before the last has been updated, the working data set contains a mixture of update stages. It may have a value in every target cell, but it is not the state defined here as the result of a complete sweep.
The distinction can be written compactly. Let \(U_X\) and \(U_Y\) denote the updates for \(X\) and \(Y\), in that order. Starting from \(Z^{(t)}\), the state after only the first update is \(U_X(Z^{(t)})\); the post-sweep state is \(Z^{(t+1)}=U_Y(U_X(Z^{(t)}))\). Saving after \(U_X\) and labeling that result iteration \(t+1\) would record a state before the sweep’s prescribed sequence is complete.
This is a bookkeeping justification, not a claim that no software could ever store an intermediate state. A workflow could store one if it deliberately defined and documented that output. For the usual saved completed data sets, using post-sweep states gives a consistent meaning to each iteration number and makes the schedule straightforward to reproduce.
Worked Example: Applying a Retention Schedule
A hypothetical run is planned to discard \(B=8\) initial sweeps, then save \(m=4\) states at interval \(L=3\). The first retained iteration is:
The four retained iteration numbers are \(t_j=8+3j\) for \(j=1,2,3,4\):
Thus the run must complete 20 sweeps to produce all four scheduled snapshots. Sweeps 1 through 8 are not retained, and sweeps between the retained iterations are also not saved. For example, iteration 12 is visited, but it is not one of the four selected states.
If this schedule were changed to \(L=1\) while keeping \(B=8\) and \(m=4\), the saved iterations would instead be 9, 10, 11, and 12, requiring 12 complete sweeps. Saving more frequently changes storage and the spacing of snapshots; it does not change what counts as a completed data set.
Intermediate States, Saved States, and Practical Choices
Three common storage choices have different meanings. Saving every post-sweep state preserves the full sequence after the initial discarded sweeps, at the cost of more storage. Saving at a fixed interval retains fewer snapshots while leaving the chain’s intervening updates intact. Saving only the final state produces one completed data set, not a collection of snapshots for multiple-imputation analysis.
| Choice | What is retained? | Main caution |
|---|---|---|
| Every sweep after \(B\) | Each complete post-sweep state in the selected range | Can require substantial storage; consecutive states are not thereby independent. |
| Every \(L\) sweeps after \(B\) | States at \(B+L,\ldots,B+mL\) | Spacing alone does not establish adequate chain behavior. |
| Final state only | One complete data set | It does not supply the multiple completed data sets needed for ordinary multiple-imputation pooling. |
The values in the imputed cells should not be averaged across iterations and then treated as a saved data set. For example, averaging iteration-specific imputations \(12.4\) and \(11.7\) gives \(12.05\), but that number is not the \(X\) imputation in either of the two chain states. More importantly, replacing stochastic draws with their averages can suppress the between-imputation variation that multiple-imputation procedures are designed to carry through analysis. The saved objects are whole data sets, each built from a particular state; estimates are calculated within each data set and combined using the appropriate pooling method.
When multiple data sets are required, a run can retain several scheduled snapshots from a chain. The interpretation of multiple chains and multiple imputations, and how to assess chain behavior, requires separate attention. In particular, a chosen \(B\) or \(L\) is a design decision, not proof that discarding or spacing states has solved a convergence problem.
Worked Example: Avoiding a Mid-Sweep Save
Suppose the \(X\)-then-\(Y\) update order is used for the same hypothetical data layout. At the end of sweep 5, let the current imputations be \(X_2=11.2\) and \(Y_3=5.4\). During sweep 6, the \(X\) update produces \(X_2=12.1\). Before the \(Y\) update has run, the working values are \(X_2=12.1\) and \(Y_3=5.4\).
That intermediate table has no blank target cells, but it mixes the new \(X\) draw from sweep 6 with the old \(Y\) draw carried forward from sweep 5. It is not the result of the specified complete sweep. Suppose the subsequent \(Y\) update produces \(Y_3=6.0\). Only then is the post-sweep-6 state available:
| Person | \(X\) | \(Y\) |
|---|---|---|
| 1 | 10 | 2 |
| 2 | 12.1 | 4 |
| 3 | 14 | 6.0 |
| 4 | 18 | 8 |
The correct record for the scheduled iteration is this post-sweep table, not the mid-sweep working table. In both tables, the observed values shown remain unchanged; the difference is whether all updates in sweep 6 have finished.
Verification and Common Failure Modes
A saved data set should be checked as a data object and as a chain record. These checks do not establish that the imputation models are adequate, but they can catch errors in saving and indexing.
- Verify the iteration label. Record the completed sweep number for every saved state and check it against \(B+jL\).
- Check completeness for target cells. Confirm that each originally missing cell designated for imputation has a value in the saved data set.
- Check observed values. Compare the observed cells in each saved data set with the original data; they should be identical when observed values are fixed.
- Check the update boundary. Save after all specified variable updates have completed, not partway through a sweep.
- Check plausible values and types. For example, a categorical variable should retain allowable categories, and a count variable should not acquire impossible fractional values unless the model and measurement scale justify them.
A frequent mistake is to treat a complete-looking data file as necessarily a correctly saved state. A file with no missing entries could still mix stages, overwrite observed values, or carry the wrong iteration label. Another mistake is to assume that spacing snapshots makes them independent. Spacing changes which states are retained; it does not itself demonstrate independence, good model specification, or adequate chain behavior.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- If \(B=5\), \(L=2\), and \(m=3\), which iteration numbers are saved, and how many complete sweeps must finish?
- At the start of a sweep, a missing \(X\) value is 7.8. The \(X\) update changes it to 8.4, but the subsequent \(Y\) update has not yet occurred. Is this the post-sweep state? Explain.
- Why is averaging an imputed cell’s values across two saved iterations not the same as saving either completed data set?
- Name two checks that can detect an error in a saved completed data set.
Solutions. (1) Apply \(t_j=B+jL=5+2j\) for \(j=1,2,3\), giving iterations 7, 9, and 11. The last retained state requires 11 complete sweeps. (2) No. The \(X\) update has finished, but the \(Y\) update has not, so the prescribed full sweep is incomplete. (3) The average is not the imputed value in either iteration-specific state; it creates a different value and can suppress variation among the stochastic imputations. Save each data set as a whole state instead. (4) For example, verify that all target missing cells have values and compare observed cells against the original data; also check the iteration labels against the schedule.