Auditing a Published Large Multi-Arm Trial Case

What an Audit Needs to Establish

A published trial’s conclusion is credible only if the design, conduct, analysis, and wording fit together. An audit does not replace a full statistical review. It checks whether the claim can be traced back to the participants who were eligible, the treatment comparisons that were actually randomized, the outcome defined in advance, and the results reported for those arms.

In a large multi-arm platform trial, this chain can be less straightforward than in a fixed two-arm study. Arms may open or close at different times, eligibility criteria or usual care may change, and participants may be randomized to comparisons that do not all share the same dates or control group. The introductory material on comparing control conditions in large multi-arm trials is relevant here: the audit should identify which control participants are genuinely concurrent with each experimental arm, rather than treating every control observation as interchangeable.

A useful audit asks five linked questions: Who entered the comparison? When did the relevant protocol apply? What outcome was primary? What happened in each randomized arm? Does the conclusion stay within the reach of those data? No single attractive result can compensate for a broken link in this chain.

A Published Case: Dexamethasone in RECOVERY

The RECOVERY platform evaluated multiple treatments for hospitalized patients with COVID-19. In its published dexamethasone comparison, 6,425 participants were randomized: 2,104 to dexamethasone and 4,321 to usual care. The primary outcome was death from any cause within 28 days after randomization. The report gave 482 deaths (22.9%) in the dexamethasone group and 1,110 (25.7%) in the usual-care group, with an age-adjusted rate ratio of 0.83 (95% confidence interval 0.75 to 0.93; \(P=0.001\)).

These figures allow several checks. First, the arm totals reconcile: \(2{,}104+4{,}321=6{,}425\). Second, the death counts reproduce the displayed percentages to rounding: \(482/2{,}104=0.2291\), or 22.9%, and \(1{,}110/4{,}321=0.2569\), or 25.7%. Third, the primary endpoint is a clinically direct, fixed-time outcome, and the reported comparison is between randomized groups. These checks support the internal coherence of the reported result; they do not, by themselves, settle every question about generalizability or subgroup interpretation.

The crude risk difference is \(0.2291-0.2569=-0.0278\), or about 2.78 percentage points fewer deaths by day 28 in the dexamethasone group. This is not the same measure as the published age-adjusted rate ratio of 0.83. A crude risk ratio from the displayed percentages would be \(0.2291/0.2569\approx0.892\). The adjustment method and estimand matter: the audit should report the study’s adjusted estimate as reported, and label any calculation from raw arm proportions as crude.

The recruitment dates and protocol history are also part of the audit. The comparison’s reported participants were recruited during the period when dexamethasone was being evaluated, with recruitment to that allocation stopping after the accumulating evidence was reviewed. For a platform study, the relevant protocol versions and dates help establish which eligibility rules, treatment choices, and follow-up procedures applied to this comparison. The flow diagram should be read for the dexamethasone comparison specifically: the total number ever randomized in the whole platform is not automatically its denominator.

The trial enrolled hospitalized patients with clinically suspected or laboratory-confirmed COVID-19. That defines the population to which the comparison most directly applies; it does not directly establish effects for people treated in the community or for every later variant, care setting, or standard of care. The primary result supports a lower 28-day mortality rate in the randomized comparison as analyzed. The clinical conclusion should not be expanded into a claim that the treatment benefits every hospitalized patient equally.

Worked Example: Reconciling a Hypothetical Flow Diagram

This kind of reconciliation is not clerical trivia. A discrepancy can reveal that a table uses a different population from the protocol-defined analysis, or simply that the paper needs a clearer explanation. The conclusion should wait until the denominators have been traced.

Deriving and Interpreting the Arm-Level Contrast

For a binary endpoint such as death by a specified day, let \(n_T\) and \(n_C\) be the numbers randomized to treatment and control, and let \(d_T\) and \(d_C\) be their observed deaths. The crude risks are \(d_T/n_T\) and \(d_C/n_C\). Subtracting the control risk from the treatment risk gives the absolute risk difference:

Under valid random allocation, a clearly defined treatment strategy, and sufficiently complete outcome ascertainment, the arm contrast estimates the effect of assignment to the treatment strategy in the randomized population. This interpretation depends on preserving the randomized groups in the analysis and on addressing missing outcomes appropriately. If treatment received differs from treatment assigned, the randomized comparison still concerns assignment, not necessarily the effect of actually adhering to treatment.

For RECOVERY’s dexamethasone comparison, the calculation is:

Thus, the crude difference is approximately 2.78 percentage points fewer deaths by day 28 in the dexamethasone arm. It is a descriptive calculation from the displayed counts. It should not be presented as the adjusted rate ratio, and the point estimate alone does not provide uncertainty. For formal inference, use the report’s prespecified analysis and confidence interval, checking that its model and analysis population match the question being answered.

Worked Example: Checking a Hypothetical Primary Endpoint

A protocol timeline helps distinguish legitimate amendments from outcome switching. Record when each version took effect, which outcomes were primary at each stage, and whether decisions preceded access to unblinded comparative results. A changed endpoint is not necessarily evidence of misconduct, but unexplained changes weaken confidence that the reported emphasis reflects the original analysis plan.

Subgroups, Diagnostics, and Failure Modes

A multi-arm report often presents results by baseline severity, age, or another characteristic. In the RECOVERY dexamethasone report, mortality results differed across baseline respiratory-support categories: the reported benefit was clearest among participants receiving invasive ventilation or oxygen, whereas the estimate among those receiving no respiratory support did not establish benefit. These are clinically relevant patterns, but subgroup estimates are less precise than the overall comparison and require careful interpretation.

A common error is to say that treatment works in one subgroup but not another because one subgroup has a statistically significant result and the other does not. That reasoning is invalid: a difference between “significant” and “not significant” is not itself evidence that the treatment effects differ. A treatment-by-subgroup interaction, interpreted with attention to prespecification, multiplicity, and uncertainty, is more relevant. Even then, subgroup evidence usually warrants a narrower conclusion than the primary randomized comparison.

Other diagnostic warning signs include:

• Arm totals, event counts, or analysis denominators that do not reconcile with the flow diagram.
• A comparison that combines control participants enrolled before an arm opened or after it closed without explaining the time structure.
• Eligibility criteria or the primary outcome that differ between the protocol and publication without a dated account of amendments.
• Different follow-up windows across arms, unexplained exclusions after randomization, or substantial missing outcome data handled without sensitivity analysis.
• A conclusion that uses relative effects without stating the outcome, time point, population, and uncertainty, or that implies a causal effect beyond the randomized comparison.

How to Make the Audit Useful

The best alternative to a narrative summary is a comparison-specific audit table: record each arm’s randomized count, analyzed count, event count, follow-up window, and reported effect estimate. A flow diagram is valuable for tracing participants, but it cannot replace the protocol timeline or the results table. Conversely, a results table without a flow check can conceal post-randomization exclusions.

A crude arm-level calculation is useful as an independent arithmetic check and for understanding absolute scale. It is not a substitute for the trial’s prespecified analysis, especially when adjustment, stratification, censoring, or time-to-event methods are used. Per-protocol analyses may address adherence-related questions, but they can lose the protection of randomization when exclusions depend on post-randomization events. For the primary treatment-assignment question, the randomized comparison is generally the starting point.

A strong audit has limits. It can establish whether the published claim is supported by the reported design and results, but it cannot recreate unavailable participant-level data or prove that every protocol decision was unbiased. It also cannot guarantee that an estimate applies to a different population, time period, or standard of care. State these boundaries explicitly rather than treating a coherent report as universally conclusive.