Common Error: Applying Reference Rules Before Dropout

Why the Dropout Boundary Matters

A reference-based imputation rule makes an assumption about outcomes that are missing after a participant discontinues the assigned treatment or stops providing follow-up. It does not authorize replacing outcomes that were actually observed before that point. Confusing these two tasks changes the data being analyzed: the result no longer represents the observed trial record completed under a missing-data assumption, but a modified record in which observed information has been discarded.

The distinction is especially important when dropout occurs at different visits across participants, or when a participant has an intermittent missing visit before later returning to provide an outcome. The dropout boundary determines which missing outcomes receive a reference-based assumption. It does not erase the distinction between an observed value and a missing one.

Definition: Let \(Y_{ij}\) be the outcome for participant \(i\) at scheduled visit \(j\), where visits are indexed \(j=0,\ldots,K\). Let \(D_i\) denote the first scheduled visit in the participant’s period of permanent post-dropout missingness. Under a monotone dropout pattern, outcomes at visits \(j\geq D_i\) are missing because of dropout. An outcome observed at any visit before \(D_i\) remains observed and is retained as recorded. If an outcome before \(D_i\) is missing intermittently, it is imputed using the separately specified strategy for that missingness; the post-dropout reference rule does not apply to it merely because it is missing.

The index \(D_i\) here marks the first post-dropout missing visit, not the last observed visit. Thus, if visits 0, 1, and 2 are observed and visit 3 is the first permanently missing visit, then \(D_i=3\). The rule’s timing is defined relative to the missing outcomes, rather than by relabeling the last observed visit as the first missing one.

For post-dropout missing outcomes, jump to reference (J2R) and copy-reference (CR) make different assumptions about the post-dropout distribution. At a high level, J2R uses a reference-arm distribution after dropout; CR assumes the post-dropout changes follow reference-arm changes, conditional on the history specified by the model. Both are assumptions for missing outcomes. Neither rule changes outcomes already observed. The predictive distributions must be specified and estimated using the appropriate reference-based model before imputations are drawn.

Separate the Observed Record From the Imputation Rules

A completed data set contains the observed values and an imputed value for every missing outcome included in the analysis. For a given participant, the completed total across visits is therefore the sum of all observed outcomes and all imputed missing outcomes. When missing visits occur both before and after dropout, the missing outcomes must be partitioned according to the strategy assigned to each part of the record.

\sum_{j=0}^{K}Y_{ij}^{\mathrm{comp}} = \sum_{j:\,Y_{ij}\ \mathrm{observed}}Y_{ij} + \sum_{\substack{j<D_i\\Y_{ij}\ \mathrm{missing}}}Y_{ij}^{\mathrm{imp,pre}} + \sum_{\substack{j\geq D_i\\Y_{ij}\ \mathrm{missing}}}Y_{ij}^{\mathrm{imp,post}}.

Here \(Y_{ij}^{\mathrm{comp}}\) is the completed value; \(Y_{ij}^{\mathrm{imp,pre}}\) is an imputation for a missing visit before the post-dropout period; and \(Y_{ij}^{\mathrm{imp,post}}\) is an imputation governed by the post-dropout reference-based rule. If a participant has no intermittent missing outcomes before dropout, the middle sum is zero. If every value is observed, both imputation sums are zero.

Worked Examples

Worked Example: Retaining an Observed Visit Under J2R

Consider a hypothetical participant assigned to the treatment arm, measured at four visits. The observed outcomes are 50 at visit 0, 56 at visit 1, and 62 at visit 2. Visit 3 is the first permanently missing visit, so \(D_i=3\). Suppose the fitted J2R predictive distribution for visit 3 has mean 55 and standard deviation 4, and one imputation draw is 53.2. The values are in a hypothetical outcome scale measured in points.

The J2R draw is used for the missing visit 3 only. Visits 0, 1, and 2 were observed, so they remain 50, 56, and 62. The completed sequence is \(50,56,62,53.2\), and the completed four-visit mean is

\bar{Y}_i^{\mathrm{comp}} = \frac{50+56+62+53.2}{4} = \frac{221.2}{4} = 55.30\ \text{points}.

The predictive mean of 55 and the draw of 53.2 describe the assumed missing visit, not an alternative value for visit 2. Replacing the observed 62 with 55 would produce a different record and a different mean, \((50+56+55+53.2)/4=53.55\) points. That calculation is not an application of J2R to the stated missingness pattern: it overwrites observed information.

Worked Example: Intermittent Missingness Before Dropout

Suppose a hypothetical participant has four scheduled outcomes. The observed values are 40 at visit 0 and 48 at visit 2. Visit 1 is missing intermittently, but the participant returns at visit 2; visit 3 is the first visit permanently missing because of dropout, so \(D_i=3\). A separately specified pre-dropout model produces an imputation of 44.5 points for visit 1. The post-dropout reference-based predictive distribution produces a draw of 43.0 points for visit 3.

The two imputations address different missingness positions and use their assigned strategies. The observed visit-2 value of 48 remains unchanged. The completed sequence is \(40,44.5,48,43.0\), giving

\sum_{j=0}^{3}Y_{ij}^{\mathrm{comp}} = (40+48)+44.5+43.0 = 88+44.5+43.0 = 175.5\ \text{points},

and the completed mean is \(175.5/4=43.875\), or 43.88 points to four significant figures. The observed sum is 88 points; the pre-dropout imputation contributes 44.5 points; and the post-dropout imputation contributes 43.0 points. Omitting the middle term would incorrectly leave the completed total short by 44.5 points.

This example also shows why “missing” is not a sufficient description for choosing an imputation rule. The visit-1 value is missing but precedes dropout and is handled under the separately specified pre-dropout strategy. Visit 3 is missing after dropout and receives the reference-based draw.

Worked Example: Auditing a Completed-Data Calculation

Consider two hypothetical treatment-arm participants measured at visits 0 through 3. Participant A has observed values 20, 24, and 28 at visits 0, 1, and 2; visit 3 is post-dropout missing, and the reference-based imputation is 23. Participant B has observed values 22 and 26 at visits 0 and 1, then drops out before visit 2. Post-dropout imputations are 25 at visit 2 and 24 at visit 3. Assume there are no intermittent missing values before either participant’s dropout boundary.

For A, \(D_A=3\), so the observed 28 at visit 2 is retained:

20+24+28+23=95\ \text{points}.

For B, \(D_B=2\), so both missing values are post-dropout imputations:

22+26+25+24=97\ \text{points}.

Across the two completed records there are eight participant-visits. The completed-data mean is

\frac{95+97}{8} = \frac{192}{8} = 24.00\ \text{points per visit}.

Now suppose an incorrect routine replaces A’s observed value of 28 with a reference-model draw of 25. The resulting total would be \(192-(28-25)=189\), and the resulting mean would be \(189/8=23.625\), or 23.63 points per visit. This difference is caused by changing an observed value, not by the stipulated post-dropout imputation. A reliable audit checks each participant’s dropout boundary and verifies that each observed value appears unchanged in every completed data set.

Why the Rule Must Start at Dropout

The logic can be stated directly. A reference-based assumption specifies a distribution for outcomes that are unobserved after dropout. If \(Y_{ij}\) is observed before dropout, its value is already available and is part of the observed data. Replacing it with a reference-model value would apply an assumption about a counterfactual missing outcome to an outcome that is not missing.

For every visit, the data status and the imputation strategy should therefore be checked in order:

Classify the visit.
Determine whether the outcome is observed, intermittently missing before dropout, or missing in the post-dropout period.

Preserve observed outcomes.
Copy each observed value into the completed data set without alteration.

Use the assigned pre-dropout procedure.
Impute an intermittent pre-dropout missing value under the separately specified model or strategy for that outcome.

Apply the reference rule after dropout.
Draw post-dropout missing values from the specified J2R or CR predictive distribution, conditional on the history required by that model.

Audit the completed record.
Confirm that all scheduled outcomes are accounted for and that no observed value has been replaced.

The assumptions needed for this procedure include correctly identifying the post-dropout period; using the intended reference arm and fitted predictive model; and drawing imputations from a distribution appropriate to the participant’s observed history and the chosen reference rule. The separate treatment of intermittent pre-dropout missingness must also be stated and justified for the analysis. These assumptions do not imply that the missing outcomes are known or that the reference model is verified by the observed data.

Diagnostics, Alternatives, and Limitations

Several checks can reveal that a reference rule has been applied too early or inconsistently:

Observed-value check: Compare observed values in each original record with the same visits in every completed data set. They should match exactly.
Boundary check: List the first permanently missing visit for each participant and verify that reference-based imputations occur only at that visit and later, where outcomes are missing.
Intermittent-missingness check: Identify missing visits before dropout explicitly. Confirm that they are included in the completed record and assigned the pre-dropout strategy, rather than silently omitted or assigned the post-dropout rule.
Accounting check: For each participant, verify that the completed record includes one value for every scheduled visit in the analysis. The partitioned sum above should equal the direct sum of the completed values.
Model-output check: Examine imputed values and trajectories for implausible distributions or coding patterns. A plausible-looking trajectory does not, by itself, establish that the correct visits received the correct rule.

A reference-based rule is not the only possible approach. A model that assumes missing outcomes are predictable from observed history under a specified missing-at-random assumption may be used for missing visits, and complete-case analysis avoids imputing outcomes but discards participants with incomplete records. These approaches make different assumptions and answer the analysis question differently; none justifies altering an observed outcome. For intermittent missingness, the strategy must be chosen explicitly rather than inferred from the fact that the participant later drops out.

The key limitation is that reference-based imputation concerns assumptions about unobserved outcomes. It cannot recover their true values from the observed data alone, and different plausible predictive models can yield different imputations. Preserving observed values prevents one avoidable source of distortion, but it does not remove uncertainty about the missing outcomes or establish that a chosen reference assumption is appropriate.

Common mistake: “The participant eventually dropped out, so I should apply the reference rule to all of that participant’s visits.” This is incorrect. The correction is to keep every observed pre-dropout outcome unchanged, apply the separate assigned procedure to any intermittent pre-dropout missing visit, and use the reference-based rule only for missing outcomes in the post-dropout period.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A participant has observed outcomes at visits 0, 1, and 2, then permanently misses visit 3. Which visit first receives the post-dropout reference rule?
A participant has an intermittent missing outcome at visit 1, returns with an observed outcome at visit 2, and permanently misses visit 3. Should the visit-1 value be imputed using the post-dropout rule?
In a completed record, the observed outcomes sum to 72 points, a missing pre-dropout outcome is imputed as 18 points, and two post-dropout outcomes are imputed as 17 and 15 points. What is the completed total?
An algorithm replaces an observed pre-dropout value of 31 points with a reference-model draw of 27 points. What is wrong, and by how much has the completed total changed?

Solutions. (1) Visit 3 is the first post-dropout missing visit, so the reference rule begins there; visits 0–2 remain as observed. (2) No. Visit 1 is missing before dropout and requires the separately specified pre-dropout strategy. Visit 3 receives the post-dropout reference-based imputation. (3) The completed total is \(72+18+17+15=122\) points. This includes the observed sum and both categories of imputed values. (4) The algorithm has overwritten an observed outcome, which a post-dropout rule does not permit. The completed total has decreased by \(31-27=4\) points solely because of that replacement.