When Interim Displays Disagree
Interim evidence rarely arrives as one complete, perfectly aligned picture. A safety summary may look concerning while exposure-adjusted rates do not; an efficacy display may appear favorable while many outcomes are not yet mature; and a dashboard may disagree with a listing because the two were produced from different data cuts. These differences are not, by themselves, proof that a display is wrong or that a treatment effect is present. They are prompts to investigate what each display measures and how it was produced.
The task in diagnosing discordant interim evidence is to identify the source and importance of the disagreement before interpreting it. A useful review separates four questions: Are the displays based on comparable populations and time periods? Are the data sufficiently complete and valid? Do the displays measure the same clinical quantity? What uncertainty remains after the checks? This is a diagnostic exercise, not a mechanical rule for turning unlike displays into a single score.
The earlier material on safety listings and summaries, interim efficacy displays, and incomplete or delayed interim data supplies the basic interpretation of those displays. Here the emphasis is on cross-checking them: aligning definitions, populations, exposure, follow-up, and data status before drawing conclusions.
A Diagnostic Sequence
Begin by writing down exactly what each display counts or estimates. For each one, record the population or analysis set, treatment group, endpoint or event definition, denominator, observation window, data cutoff, and handling of missing or pending information. A mismatch on any of these dimensions can create apparent disagreement without any underlying contradiction.
Check whether the displays refer to the same endpoint, severity threshold, treatment group, time origin, and event-attribution rules. Similar labels do not guarantee identical definitions.
Compare randomized participants, treated participants, participants with outcome assessments, and person-time at risk. Confirm that the denominator suits the quantity being displayed.
Review visit completion, event reporting delays, pending adjudication, missing outcomes, and the timing of each extract. A newer-looking result is not necessarily more complete.
Recalculate a summary from its underlying records where feasible, then trace discrepant records through coding, deduplication, and data transformations.
Separate verified facts from plausible explanations and unresolved questions. Do not make an unexplained discrepancy disappear by selecting whichever display is more reassuring.
This sequence is deliberately diagnostic. It supports a clear account of what the evidence can and cannot establish, while leaving a committee to deliberate under its charter and trial procedures.
Counts, Rates, and Unequal Exposure
A raw event count answers how many events were recorded, not how often events occurred relative to time at risk. When exposure differs between groups, rates can help describe the difference. For a group with \(d\) events over \(T\) person-years, the incidence rate is \(d/T\). This rate is interpretable only if the event definition and time-at-risk rules are consistent across groups.
For a rough comparison, suppose event counts in two groups are independent Poisson counts, with a constant event rate within each group over the observed person-time. If the rates are \(\lambda_1=d_1/T_1\) and \(\lambda_0=d_0/T_0\), their estimated rate ratio is \(\widehat{RR}=\lambda_1/\lambda_0\). For sufficiently large event counts, the log rate ratio has approximate standard error \(\sqrt{1/d_1+1/d_0}\). This approximation describes sampling uncertainty under the stated model; it does not address confounding, event clustering, differential ascertainment, or whether a rate ratio is the most clinically relevant summary.
The approximation follows from the Poisson variance assumption. If \(D\) is Poisson with mean \(\lambda T\), then \(\operatorname{Var}(D)=\lambda T\). For a positive count, the delta method gives \(\operatorname{Var}(\log D)\) approximately \(\operatorname{Var}(D)/[E(D)]^2=1/E(D)\). Since \(\log(\widehat{RR})=\log(d_1)-\log(T_1)-\log(d_0)+\log(T_0)\), and the exposure times are treated as fixed, independent counts give approximate variance \(1/d_1+1/d_0\). This is a rough interval calculation, not a substitute for the trial’s prespecified safety analyses.
Worked Example: More Events, but Also More Exposure
Hypothetical data: An interim safety display records 12 events over 160 person-years in the investigational group and 5 over 110 person-years in the control group. The event definitions and time-at-risk rules are confirmed to be the same. Compare the crude counts and rates, then quantify the uncertainty in the rate ratio using the approximate Poisson method.
The crude counts are 12 versus 5, which might initially look concerning for the investigational group. The rates per 100 person-years are \(12/160\times100=7.500\) and \(5/110\times100=4.545\). The estimated rate ratio is \[ \widehat{RR}=\frac{12/160}{5/110}=1.650. \] Thus, the observed event rate is higher in the investigational group in this hypothetical comparison, although the counts alone do not show how uncertain that comparison is.
Under the independent Poisson and constant-rate assumptions, the approximate standard error of the log rate ratio is \[ SE\{\log(\widehat{RR})\}=\sqrt{\frac{1}{12}+\frac{1}{5}}=\sqrt{0.2833}=0.5323. \] Since \(\log(1.650)=0.5008\), the approximate 95% interval on the log scale is \(0.5008\pm1.96(0.5323)=(-0.5425,\ 1.5441)\). Exponentiating the endpoints gives an approximate rate-ratio interval of \((0.581,\ 4.684)\).
The interval is wide and includes 1. The data therefore do not precisely distinguish a lower rate from a substantially higher rate under this approximate model. This is not evidence that the groups are equivalent, nor does it settle the clinical importance of the events. The next checks include event seriousness and timing, recurrent-event patterns, potential differences in ascertainment, and whether the exposure calculation is correct. With few events or clustered events, the approximation may be unreliable.
Outcome Maturity and Missing Data
An efficacy display can look different depending on who has had enough follow-up for an outcome to be observed. A complete-case proportion, for example, describes participants with recorded outcomes; it need not describe all randomized participants if missingness or maturity differs between groups. The randomized denominator remains important for an intention-to-treat comparison, but unresolved outcomes cannot simply be treated as observed failures or successes without stating that assumption.
Worked Example: Equal Observed Response, Unequal Outcome Availability
Hypothetical data: At an interim cutoff, 24 of 80 assessed participants in the investigational group have responded; 20 of its 100 randomized participants do not yet have an assessed outcome. In the control group, 27 of 90 assessed participants have responded, while 10 of 100 randomized participants lack an assessed outcome. Assume for this illustration that response is binary and all randomized participants are intended for the comparison.
Among participants with assessed outcomes, the response proportions are \(24/80=0.300\) and \(27/90=0.300\). They are equal as complete-case summaries. But availability differs: 20% of the investigational group and 10% of the control group lack an assessed outcome. The equal observed proportions therefore do not establish that the randomized groups have equal response probabilities.
A simple range calculation shows the uncertainty due only to the missing binary outcomes, without making a missing-at-random assumption. If every missing investigational outcome were a nonresponse, its randomized-denominator response proportion would be \(24/100=0.240\); if all 20 were responses, it would be \((24+20)/100=0.440\). The corresponding control range is \(27/100=0.270\) to \((27+10)/100=0.370\).
The possible difference (investigational minus control) ranges from \(0.240-0.370=-0.130\) to \(0.440-0.270=0.170\). These broad bounds include differences in either direction. They are not a formal efficacy analysis, but they show why equal complete-case proportions can be misleading when outcome availability differs. Before interpretation, check why outcomes are unavailable, how many are merely pending because of follow-up timing, and what the prespecified analysis does with missing data. Do not replace the randomized comparison with an analysis restricted to participants who received a particular amount of treatment: exposure after randomization can be related to prognosis and tolerability.
Data-Quality Displays That Conflict
A discrepancy between a dashboard and a line listing is a data-quality question before it is a clinical interpretation question. Common causes include different extraction times, different event cutoffs, duplicate reports, coding changes, different seriousness or severity filters, and different rules for including ongoing events. A count mismatch should be traceable to records and processing rules; an unexplained difference should remain visible as unresolved.
Worked Example: Reconciling Two Safety Counts
Hypothetical data: A dashboard reports 9 serious events in the investigational group, while a listing prepared for the same interim review appears to contain 11. The listing’s header shows that it was extracted one day later. On record-level review, one additional event was reported after the dashboard extract, and one of the remaining listing rows is a follow-up report for an event already counted.
Start with the apparent difference: \(11-9=2\) rows. The later report explains one additional unique event, while the follow-up row does not represent a new event. After removing the duplicate follow-up row, the listing contains 10 unique events: the 9 available at the dashboard cutoff plus 1 reported later. Thus, the two displays differ because their cutoffs differ, not because 2 additional unique events were identified.
This reconciliation does not establish whether the later event changes the clinical interpretation. That requires checking the event details, treatment group, seriousness criteria, timing, and any applicable reporting or review procedures. If the record cannot be confidently classified as a new event or follow-up, document the uncertainty and request verification rather than forcing the count to agree.
How to Interpret Cross-Display Tensions
Displays can disagree for substantive as well as operational reasons. A safety rate may rise while an efficacy estimate also appears favorable; that is a possible benefit-risk tension, not a mathematical contradiction. Conversely, safety counts can rise simply because one group has accumulated more person-time. Exposure summaries help explain opportunity for events, but exposure itself is post-randomization and should not automatically replace randomized-group comparisons.
| Approach | What it helps with | Main limitation |
|---|---|---|
| Read each display separately | Clarifies what each summary says on its own terms | Can miss denominator, cutoff, and maturity mismatches between displays |
| Use a structured cross-check | Traces differences in definitions, populations, exposure, follow-up, and data processing | May identify unresolved uncertainty rather than produce one definitive estimate |
| Choose a preferred display and disregard the rest | May appear to simplify discussion | Risks hiding a meaningful discrepancy or selectively privileging reassuring evidence |
| Adjust comparisons informally for exposure or missingness | Can prompt useful sensitivity questions | Can introduce bias if post-randomization selection or unsupported assumptions are treated as definitive |
A careful account distinguishes what has been verified from what is inferred. For example: “The higher count is explained in part by greater person-time; the exposure-adjusted rate remains numerically higher, but the interval is imprecise.” Or: “Observed response proportions agree among assessed participants, but outcome availability differs, so the interim comparison is not robust to unresolved outcomes.” These statements preserve both the signal and its limitations.
Failure Modes and Practical Checks
- Comparing incompatible denominators. A percentage among treated participants and a percentage among randomized participants answer different questions. Label each denominator explicitly.
- Treating person-time adjustment as a cure-all. Rates account for differing time at risk under their definitions; they do not fix differential event detection, informative treatment discontinuation, or confounding.
- Calling pending outcomes negative. Pending is a data status, not an outcome value. Report maturity and apply the prespecified missing-data approach.
- Assuming a later extract is complete and correct. New records may be added, but later extracts can also contain duplicates, recoding, or unresolved entries. Verify the record trail.
- Replacing the randomized comparison with an exposed-only comparison. Participants who start, continue, or tolerate treatment may differ systematically from those who do not. Describe exposure, but do not interpret a selected subgroup as if randomization were preserved.
- Equating lack of precision with no risk. A wide interval that includes the null can still be compatible with clinically important benefit or harm. State the range of uncertainty.
The diagnostic process has limits. Interim summaries may be too sparse to distinguish chance variation from a real pattern. Definitions can be unsettled, adjudication may be pending, and data-quality review may not resolve every record before deliberation. Some concerns require urgent handling under the trial’s established safety and governance procedures; diagnosing discordance is not a reason to delay such escalation. Record unresolved issues and their possible implications rather than implying that further checks will necessarily settle them.
Check Your Understanding
For each question, identify the diagnostic issue and compare your reasoning with the solution immediately below it.
- A safety summary has 8 events over 80 person-years in one group and 7 events over 140 person-years in another. Calculate both rates per 100 person-years. Why do the counts alone give a different impression?
- An efficacy display reports 40 responses among 100 assessed participants in one group and 36 among 90 assessed participants in another. In the first group, 20 randomized participants remain unassessed; in the second, 10 remain unassessed. What can and cannot be concluded from the observed proportions?
- A dashboard reports 6 unique events and a listing has 8 rows. What should be checked before treating the difference as two additional events?
- Why is an analysis limited to participants who received at least four treatment cycles not automatically a valid replacement for a randomized-group efficacy comparison?
Solutions. (1) The rates are \(8/80\times100=10.00\) and \(7/140\times100=5.000\) events per 100 person-years. The group with more events also has less exposure, so counts alone conceal the difference in time at risk; definitions and ascertainment still need checking. (2) Both observed proportions are \(40/100=0.400\) and \(36/90=0.400\), but outcome availability is 80% versus 90%. The observed proportions agree among assessed participants, not necessarily among all randomized participants; pending outcomes and the planned missing-data analysis matter. (3) Check extraction times, event definitions, unique event identifiers, follow-up reports, filters, coding, and the record-level trail. Rows are not necessarily distinct events. (4) Receiving four cycles is a post-randomization condition associated potentially with prognosis, tolerability, or early outcome. Restricting to that subgroup can break the comparability created by randomization, so exposure descriptions do not automatically replace the randomized comparison.