Why Data Quality Is Part of Interim Interpretation
Interim reports describe data available by a particular cutoff, not necessarily all outcomes that have occurred or all outcomes that will eventually be observed. A difference between treatment groups may reflect a genuine clinical pattern, but it may also arise because outcomes are missing, reports have arrived at different speeds, events have been coded inconsistently, or participants have had unequal opportunities for follow-up. The IDMC’s task is not to repair the database during its review; it is to identify which conclusions the available data can and cannot support.
A useful diagnosis separates four questions: Are the relevant participants represented? Have their outcomes had comparable opportunities to be observed and reported? Do the records mean the same thing across groups and sites? Are the denominators appropriate for the comparison? These checks complement the clinical and statistical review of safety and efficacy; they do not replace it.
Start With the Cutoff, Population, and Denominator
Before interpreting a table, establish which participants it covers, the data cutoff, the outcome definition, and the denominator used. Check whether the denominator is all randomized participants, participants with a recorded outcome, participants exposed to treatment, or accumulated follow-up time. Each can answer a different descriptive question. A percentage without its numerator, denominator, and cutoff is difficult to audit and easy to misread.
For a binary outcome, let \(N_g\) be the number in group \(g\) for whom the outcome is required by the analysis definition, \(m_g\) the number with missing outcomes, and \(y_g\) the number with the outcome among those observed. The observed-case proportion is \(y_g/(N_g-m_g)\), while completeness is \((N_g-m_g)/N_g\). The observed-case proportion describes participants whose outcomes are recorded; it is not automatically an unbiased estimate of the proportion among all \(N_g\) participants.
Worked Example: Differentially Missing Efficacy Outcomes
Suppose a hypothetical interim report includes 100 randomized participants in each group. At the cutoff, a binary efficacy outcome is recorded for 70 participants receiving the experimental treatment, with 42 successes, and for 90 participants receiving control, with 45 successes. Assume the outcome is due for assessment for all 200 participants. The report does not establish why the 40 outcomes are missing.
The observed-case success proportions are \(42/70=0.600\), or 60.0%, in the experimental group and \(45/90=0.500\), or 50.0%, in control. Completeness, however, differs: \(70/100=70.0\%\) versus \(90/100=90.0\%\). The observed difference is 10.0 percentage points, but the unequal completeness makes a direct interpretation vulnerable to how missing outcomes differ from observed outcomes.
A simple worst-to-best bound shows how little can be concluded without assumptions about those missing outcomes. In the experimental group, the all-participant success proportion could range from \(42/100=42.0\%\), if none of the 30 missing outcomes were successes, to \((42+30)/100=72.0\%\), if all were successes. The control range is \(45.0\%\) to \(55.0\%\). Thus the experimental-minus-control difference could range from \(42.0\%-55.0\%=-13.0\) to \(72.0\%-45.0\%=27.0\) percentage points. These bounds are deliberately broad, not a confidence interval. They demonstrate that the observed 10.0-point difference does not determine even the direction of the all-participant difference.
The diagnostic conclusion is not that the treatment is ineffective or effective. It is that outcome completeness differs by 20 percentage points, and conclusions about the full randomized groups depend on assumptions or additional follow-up. The report should show missingness by group and, where available, reasons and timing.
Distinguish Missing Outcomes From Delayed Event Reporting
A missing outcome has no usable value in the analysis dataset at the cutoff. A delayed event report may refer to an event that has already occurred but is not yet captured in that dataset. Delays are especially important for safety summaries: recently enrolled participants may have less time for an event to occur, and events that have occurred may not yet have been submitted, reviewed, or coded.
Compare groups and sites using a common time basis. Useful checks include the number of participants with follow-up due, the number with source documentation, the number of events entered into the trial system, and the elapsed time between event occurrence and report entry. If source records or established reporting systems are available, reconcile a defined sample against the analysis dataset. A reporting-completeness proportion can be calculated as events entered by the cutoff divided by eligible events identified in the specified source records. Its interpretation depends on the source’s coverage and the definition of an eligible event; it is not a universal estimate of all unreported events.
Worked Example: Comparing Event-Reporting Completion
In a hypothetical reconciliation, a prespecified review of available source records identifies 25 eligible serious events in the experimental group and 20 in control by the cutoff. The trial event dataset contains 20 and 19 of these events, respectively. Assume the source review used the same event definition and cutoff in both groups, and every identified event can be matched or classified.
The proportion of source-identified events present in the trial dataset is \(20/25=80.0\%\) for the experimental group and \(19/20=95.0\%\) for control. The corresponding unmatched counts are 5 and 1. These figures indicate a group difference in reconciliation completion within the reviewed records. They do not establish that exactly five and one events are missing from the entire trial: the source review may not capture every event, and the sample or record coverage may be incomplete.
The appropriate follow-up is to examine whether unmatched events cluster by site, event date, or reporting route, and whether the same lag window was allowed in both groups. Until that review is resolved, the observed event counts may not be comparably complete. Simply dividing the reported counts by randomized sample size would not correct the reporting delay.
Check Coding and Event Definitions
Coding inconsistencies occur when clinically similar descriptions are represented differently, when coding conventions change across sites or time, or when one event is entered more than once. Conversely, distinct events can be incorrectly collapsed into a single record. Such inconsistencies can distort summaries by event category even when the underlying clinical narratives are accurate.
Check that the report states the event definition, coding version or convention, and rules for counting recurrent events. Review unexpected concentrations of related terms, abrupt changes around coding updates, duplicate participant-event records, and differences in coding patterns across treatment groups or sites. Where categories are clinically combined for review, retain enough detail to trace the grouped records back to source descriptions. A coding mismatch should be investigated; it should not be silently recoded during interpretation.
Worked Example: A Coding Mismatch in an Event Summary
Suppose a hypothetical report groups events under a prespecified respiratory category. A reconciliation finds 8 records in that category and 3 additional records described in source text as “acute breathing difficulty” but assigned to a different category at one site. Assume a blinded clinical and coding review confirms that all 3 meet the report’s stated category definition, and that these are distinct events rather than duplicates.
The reconciled count for the category is \(8+3=11\), not 8. If the relevant exposure denominator is 500 participant-years, the descriptive rate changes from \(8/500\times100=1.60\) to \(11/500\times100=2.20\) events per 100 participant-years. The difference is \(2.20-1.60=0.600\) events per 100 participant-years. This arithmetic illustrates why category consistency matters; it does not establish that the revised rate is statistically or clinically different.
The report should identify the three records as a coding discrepancy, document the review basis, and distinguish the original table from any sensitivity summary. If the records were not confirmed against the definition, adding them to the category would be unjustified.
Assess Follow-Up Balance Before Comparing Safety Counts
Participants enrolled earlier generally have more time in which events can be observed than participants enrolled later. If follow-up differs between groups, raw event counts can be misleading. For an event count \(E_g\) and accumulated time at risk \(T_g\), a descriptive rate is \(E_g/T_g\), often scaled to a convenient unit such as 100 participant-years. This comparison assumes the exposure-time calculation and event ascertainment are sufficiently consistent to make the rates interpretable. It does not by itself adjust for differences in participant characteristics, event timing, censoring, or other factors.
For a fixed-time efficacy outcome, unequal follow-up can instead mean that some participants have not yet reached the assessment window. Those outcomes should not automatically be classified as failures or treated as if follow-up were equal. Show follow-up distributions and the number due for assessment, and distinguish “not yet due” from “due but missing.”
Worked Example: Event Counts With Unequal Exposure
In a hypothetical safety summary, the experimental group has 12 events over 600 participant-years, and control has 10 events over 400 participant-years. Assume the event definitions are consistent, the time at risk is calculated by the same rule in both groups, and the data cutoff is common.
The experimental-group descriptive rate is \(12/600\times100=2.00\) events per 100 participant-years. The control rate is \(10/400\times100=2.50\) events per 100 participant-years. Although the experimental group has more events in total, its rate per observed time is lower in this summary. That observation does not prove a protective treatment effect: the comparison is descriptive, and these data alone do not address random variation, differences in who remains under observation, or reporting completeness.
A useful report would display both counts and exposure time, explain how exposure time was defined, and show whether follow-up or event reporting differs materially across groups. If only the counts were shown, a reader could incorrectly infer that the experimental group has greater risk simply because 12 exceeds 10.
A Structured Diagnostic Review
A consistent sequence helps prevent data-quality concerns from being lost in a discussion of apparent treatment differences. The checks below are diagnostic: they flag uncertainty for clarification and do not prescribe a trial decision.
Record the cutoff, eligible population, outcome definition, and denominator. Confirm whether each participant is due for the outcome or still within the expected follow-up period.
Show counts and percentages with outcomes available, by group and relevant site or time strata. Separate not-yet-due outcomes from outcomes that are overdue or otherwise missing.
Compare source records with the analysis dataset where feasible, review event-entry lags, and investigate unmatched, duplicated, or inconsistently classified records.
Re-express summaries using appropriate denominators and transparent assumptions. For missing outcomes, show plausible sensitivity analyses or bounds rather than hiding the assumptions in a single estimate.
Describe the likely direction and possible importance of each limitation only when supported by evidence. Identify unresolved items and avoid presenting a provisional correction as verified data.
Justifying Missing-Outcome Bounds
The bounds in the first example follow directly from the fact that an unobserved binary outcome can only be 0 or 1. If \(y\) successes are observed among \(N-m\) participants and \(m\) outcomes are missing, the smallest possible all-participant success proportion occurs when all missing outcomes are 0; the largest occurs when they are all 1.
No missingness assumption is needed for this range, but it may be wide. A narrower estimate requires additional assumptions, such as missing outcomes having a specified relationship to observed characteristics or outcomes. Those assumptions should be stated and assessed, not treated as facts merely because a model can be fitted. The bound is a diagnostic tool for checking whether the conclusion is robust to the unavailable data.
Comparison, Failure Modes, and Limits
| Approach | What it helps answer | Main limitation |
|---|---|---|
| Observed-case summary | What was recorded among participants with available outcomes? | Can misrepresent the full group when completeness differs or missingness is related to outcomes. |
| All-randomized denominator | What proportion of all randomized participants has a recorded outcome or event? | Does not make missing outcomes known, and may be unsuitable for exposure-based event rates. |
| Exposure-time rate | How many events were recorded per unit of observed time at risk? | Can obscure differences in participant mix, event timing, censoring, or ascertainment. |
| Source reconciliation and sensitivity bounds | Could unreported records or missing outcomes materially change the summary? | Depends on source coverage, matching rules, and the assumptions defining the sensitivity analysis. |
A common mistake is to treat an unrecorded outcome as if it were a negative outcome, or to divide event counts by the number randomized and call the result a risk when follow-up differs. The correction is to identify what is known, what is missing, what is not yet due, and what denominator matches the quantity being summarized. Another pitfall is to interpret a coding or reconciliation difference as evidence of a treatment-group difference before checking whether the discrepancy is group-specific or merely a reporting artifact.
These diagnostics have limits. A clean reconciliation of available records cannot prove that no events are unreported. A broad missing-outcome bound may be too uninformative to settle an interpretation. Rates do not eliminate bias from unequal or informative follow-up. Finally, an interim report may be based on immature data by design; the important distinction is whether the immaturity is expected, quantified, and comparable, rather than whether every record is complete.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
- A group has 80 participants due for a binary outcome; 60 outcomes are recorded. What is completeness, and how many outcomes are missing?
- Why is an event that has not yet been entered into the trial dataset not necessarily the same as an event that has not occurred?
- A report gives 9 events over 300 participant-years. Calculate the descriptive rate per 100 participant-years.
- Why can a high observed-case success proportion be misleading when outcome completeness is lower in one group?
Solutions. (1) Completeness is \(60/80=75.0\%\); 20 outcomes are missing. (2) The event may have occurred and be documented in a source record but not yet submitted or entered by the cutoff; reporting delay is distinct from absence of the event. (3) \(9/300\times100=3.00\) events per 100 participant-years, interpreted descriptively under the stated time-at-risk definition. (4) If missing outcomes differ from observed outcomes, the observed participants may not represent the full group; unequal completeness can also make the groups’ observed-case proportions less comparable.