Protecting Selection Decisions From Data-Quality Problems

Why Interim Data Quality Needs a Decision Rule

An interim selection rule can be calculated exactly as planned and still lead to a poor decision if the data feeding it are incomplete, inconsistent or wrongly coded. In a multi-arm multi-stage (MAMS) trial, the problem is consequential: an arm may be dropped or continued on the basis of a score, so a small number of delayed or misclassified outcomes can affect which treatments remain under study.

Data-quality checks are not a substitute for the statistical selection rule. They establish whether the interim analysis dataset is fit to be passed to that rule. The gate should be specified before the interim data are examined, applied consistently across arms, and documented. It should not be improvised after analysts see which arm a questionable record might help.

Specify the Dataset and the Gate

Before checking records, define the interim snapshot: the calendar cutoff, the participant population eligible for this look, the endpoint and assessment window, the required data fields, and the version of the selection analysis to which the data will be supplied. The eligible set must be defined from trial rules, not reconstructed from whichever outcomes happen to be available. For participant \(i\) in arm \(k\), write the endpoint as \(Y_{ik}\); the score for arm \(k\), denoted \(S_k\), is calculated by the prespecified selection rule only after the gate passes.

A useful completeness specification accounts for each eligible participant at the cutoff. For example, each participant may have a validated endpoint, an outcome that is pending within its allowed assessment window, or a documented status that prevents ascertainment. The categories and their implications must be defined in advance. In particular, “no value in the analysis file” is not a sufficient status: it could mean a visit is pending, a record has not transferred, a participant withdrew, or a value was excluded during data processing. The earlier material on missing interim outcomes addresses how missingness enters selection; here the concern is whether the data snapshot and its status information are trustworthy enough to use.

Completeness thresholds should reflect the selection procedure and trial operations. A protocol might require a minimum number of validated outcomes per arm, a minimum proportion of eligible outcomes with validated status, or both. It might also require that no unresolved discrepancy remains for a field that can alter eligibility, endpoint value, arm assignment or endpoint timing. There is no universal percentage that makes an interim dataset acceptable. The threshold and the action on failure—such as delaying the analysis, obtaining data, or applying a separately specified procedure—must be set before results are reviewed.

Validation checks should likewise be defined before unblinding of arm-specific scores. They commonly cover participant identity and duplicate records; randomized arm assignment; outcome definition and coding; assessment date and window; permitted value range; units and transformations; logical consistency among related fields; and agreement with the designated source record. Automated checks can find many problems, but a flagged value is not automatically wrong: it must be queried and resolved or handled according to the prespecified rule. Maintain an audit trail recording the original value, the query, the resolution, who made it and when.

Why Delayed or Incorrect Records Can Change a Score

Consider a mean outcome in one arm. If \(n\) eligible participants are intended to contribute, \(n-m\) outcomes are available at the cutoff, and the \(m\) delayed outcomes would have mean \(\bar{Y}_{\mathrm{delayed}}\), then the full-set mean would be

The available-case mean differs from the full-set mean by

Thus, even a modest fraction of delayed outcomes can matter if their values tend to differ from those already available. The direction and size of the difference cannot be inferred from the mere fact that records are delayed. This is a reason to apply a consistent, planned completeness rule—not a reason to assume that pending outcomes favor or disadvantage a particular arm. A coding error can be more direct: it changes the recorded value itself, and therefore can change the score even when every participant appears to have an outcome.

Check Values Against Their Sources

A validation gate should address errors capable of changing the analysis, rather than merely checking whether a file opens. For a continuous endpoint, check the recorded unit, measurement window, permissible range and any unit conversion. For a binary endpoint, check the event definition, coding direction and source evidence for the event status. For either type, confirm that participant identifiers map uniquely to the randomized allocation and that duplicate, out-of-window or post-cutoff records have not entered the snapshot.

A useful diagnostic is to compare distributions and data-status summaries across arms, but a difference is a signal to investigate, not proof of an error. For instance, a higher fraction of pending outcomes in one arm might reflect operational delays, different visit timing or an extraction problem. It can also have consequences for an available-case score. The team should check the source and process that generated the pattern before deciding whether the gate is passed. Do not “correct” a genuine outcome simply because it looks unusual.

Responding to a Failed Gate

A failed gate should lead to a predefined operational response, not an improvised statistical adjustment. Depending on the protocol, the team may delay the analysis until a specified data cutoff, correct and re-freeze the dataset after resolving queries, or use a preplanned method that explicitly accommodates the relevant data status. That last option requires its own assumptions and must not be invented after seeing the arm scores. The record of the interim process should identify the snapshot, checks performed, failures, resolutions and the authorized dataset version.

A hard gate that waits for a common, specified cutoff is easy to audit, but can delay a decision and does not ensure that all values are accurate. Arm-specific completeness thresholds may be operationally necessary, but they can permit different amounts of information across arms; the rule must therefore make clear how the selection procedure handles the resulting sample sizes. Automated range and consistency checks are efficient, but cannot establish that a source value was measured correctly. Direct source verification is stronger for selected fields but takes resources and may not be feasible for every record. These approaches are complementary rather than interchangeable.

Common Mistakes and Limitations

• Using “complete cases” as the definition of completeness. This ignores eligible participants whose outcomes are pending or whose status has not been reconciled. Define the denominator and status categories explicitly.
• Treating a numerical range as proof of validity. A plausible value can still be assigned to the wrong participant, arm, endpoint window or unit. Range checks need identity and source checks as appropriate.
• Resolving a query in the direction that improves an arm. Values must be corrected from verified evidence under a uniform rule. Outcome-driven editing undermines the selection process.
• Assuming equal record counts mean equal data quality. Two arms may have the same number of outcomes but different unresolved discrepancies, assessment timing or source verification.
• Changing a threshold after looking at scores. This makes the gate depend on the result it is meant to qualify. Prespecify both the criteria and what happens when they fail.

A quality gate cannot remove all measurement error, establish that a clinical endpoint is meaningful, or guarantee that an observed interim effect will persist. Its purpose is narrower and practical: to prevent a selection rule from acting on a dataset that fails defined conditions for completeness and reliability. The criteria should be proportionate to the endpoint, the consequences of dropping an arm and the time available for interim review.