RECOVERY Hydroxychloroquine Futility Decision

Why the Hydroxychloroquine Decision Matters

An adaptive trial can stop assigning participants to one treatment while continuing to study other treatments. The key question is what the available evidence justifies saying about the treatment that closed. In RECOVERY, recruitment to the hydroxychloroquine comparison stopped after interim evidence indicated no meaningful benefit for hospitalized patients. That decision was a judgment about whether continuing recruitment was worthwhile; it was not, by itself, proof that hydroxychloroquine and usual care were equivalent.

RECOVERY was a randomized platform trial evaluating treatments for patients hospitalized with COVID-19. On 5 June 2020, recruitment to its hydroxychloroquine comparison stopped following advice from the independent data monitoring committee. The subsequent report gives the final comparison: 421 of 1,561 participants assigned hydroxychloroquine died within 28 days, compared with 790 of 3,155 assigned usual care. The reported mortality rate ratio was 1.09, with a 95% confidence interval of 0.97 to 1.23 and a p-value of 0.15.

Those final results help explain the decision, but they must not be mistaken for the exact interim data available to the committee on the stopping date. Nor does a nonsignificant p-value mean that the treatments were proven identical. A careful account separates the operational decision to stop recruitment from the statistical claims supported by the observed data.

Futility Is a Decision About Continuing

Definition: A futility decision is a decision to stop or modify recruitment because the evidence available at an interim review suggests that continuing the study is unlikely to achieve its relevant objective, such as demonstrating a clinically worthwhile benefit. Futility is not a finding that the treatment effect is exactly zero.

A futility assessment depends on the objective and the design. The target might be a statistically significant benefit, a benefit of at least a specified clinical size, or some other prespecified goal. A common design-based measure is conditional power: the probability of meeting the trial’s success criterion at the planned final analysis, conditional on the interim data and on specified assumptions about what happens after the interim look.

Conditional power is not a single number determined by the interim data alone. It depends on assumptions about the future treatment effect, the remaining information, and the planned final analysis. For example, it may be calculated assuming the interim estimated effect persists, or assuming a target effect used in the original design. If those assumptions lead to very different answers, a single conditional-power value should not be treated as decisive.

The RECOVERY decision should be described at the level supported by the available account: recruitment to hydroxychloroquine stopped after the committee considered interim evidence to show no beneficial effect. This does not establish that a particular conditional-power threshold was used. Unless a specific prespecified rule and its inputs are documented, it is not appropriate to retrofit one onto the decision.

Worked Example: A Hypothetical Conditional-Power Review

Suppose a different trial has a one-sided benefit test with a final critical value of 1.96 on a standardized statistic \(Z\), where positive values favor treatment. At an interim analysis, half the planned information has accrued, so \(t=0.50\), and the observed interim statistic is \(z=-1.00\). For illustration, assume the design alternative corresponds to a final noncentrality parameter \(\delta=2.50\). Calculate conditional power under this assumed alternative.

Under the normal independent-increments approximation, the final statistic conditional on the interim result has mean \(\sqrt{t}z+\delta\sqrt{1-t}\) and variance \(1-t\). Thus:

Z_{\mathrm{final}}\mid z \ \approx\ N\!\left(\sqrt{t}z+\delta\sqrt{1-t},\,1-t\right).

Substituting the stated values gives a conditional mean of \(\sqrt{0.50}(-1.00)+2.50\sqrt{0.50}=1.061\), and a conditional standard deviation of \(\sqrt{0.50}=0.7071\). Therefore:

P(Z_{\mathrm{final}}>1.96\mid z) =1-\Phi\!\left(\frac{1.96-1.061}{0.7071}\right) \approx 1-\Phi(1.271) \approx 0.102.

Under these assumptions, the chance of crossing the success boundary is about 10.2%. If the protocol had specified a nonbinding futility rule at conditional power below 20%, this calculation would meet that rule. But changing the assumed future effect changes the result; this is an illustration of a possible method, not a description of RECOVERY’s documented stopping rule.

What the RECOVERY Mortality Counts Show

The final counts permit a transparent, approximate calculation. For this illustration, treat each participant’s 28-day mortality outcome as a binary event and calculate the unadjusted risk ratio. The calculation assumes independent participant outcomes within randomized groups and uses the large-sample approximation for the log risk ratio. It does not reproduce every feature of the trial’s reported analysis.

Definition: For treatment and control event counts \(a\) and \(c\), with group sizes \(n_T\) and \(n_C\), the sample risk ratio is \(\widehat{RR}=(a/n_T)/(c/n_C)\). An approximate standard error for its logarithm is \(\sqrt{1/a-1/n_T+1/c-1/n_C}\). A 95% confidence interval is obtained by forming \(\log(\widehat{RR})\pm1.96SE\) and exponentiating both limits.

Worked Example: An Approximate Risk Ratio From the Final Counts

Use the final 28-day deaths: 421 among 1,561 participants assigned hydroxychloroquine and 790 among 3,155 assigned usual care.

The observed risks are \(421/1561=0.2697\), or 26.97%, and \(790/3155=0.2504\), or 25.04%. The unadjusted risk ratio is:

\widehat{RR} =\frac{421/1561}{790/3155} \approx 1.077.

In these groups, the observed 28-day mortality risk was about 1.077 times as high with hydroxychloroquine assignment as with usual care. The absolute difference was \(0.2697-0.2504=0.01930\), or 1.930 percentage points higher in the hydroxychloroquine group.

For the approximate log-scale standard error:

SE =\sqrt{\frac{1}{421}-\frac{1}{1561}+\frac{1}{790}-\frac{1}{3155}} =\sqrt{0.002684} \approx 0.05180.

Using \(\log(1.0771)\approx0.0743\), the 95% interval on the log scale is \(0.0743\pm1.96(0.05180)\), or approximately \((-0.02727,0.1758)\). Exponentiating gives an approximate risk-ratio interval of \((0.9731,1.192)\). This interval includes 1, so the unadjusted calculation does not show a statistically significant difference at the two-sided 5% level. It also includes values below 1, as well as values representing higher mortality with hydroxychloroquine.

This calculation is illustrative, not a replacement for the trial’s reported result. The published rate ratio of 1.09 and its 95% confidence interval of 0.97 to 1.23 come from the trial’s own analysis. Differences between the simple calculation and the published estimate can arise because the methods and estimands need not be identical.

Futility Is Not Equivalence

A conventional test of no difference asks whether the data provide sufficient evidence against a null hypothesis such as \(H_0: RR=1\). Failure to reject that null does not establish it. Equivalence asks a different question: whether the treatment effect is sufficiently close to no difference according to a prespecified clinical margin.

Definition: An equivalence margin specifies the range of treatment effects considered clinically negligible. For a risk ratio, an equivalence claim requires a prespecified interval around 1 and evidence that the treatment effect lies entirely within that interval, using a valid analysis and the planned error control.

For example, if a trial prespecifies equivalence as \(0.90<RR<1.10\), it must show that the uncertainty interval required by the chosen equivalence procedure falls wholly inside those limits. A 90% confidence interval is commonly used for two one-sided tests at a 5% level, provided the design and analysis support that procedure. Merely obtaining a p-value above 0.05 for a test of \(RR=1\) is not an equivalence analysis.

Worked Example: Why a Nonsignificant Result Does Not Prove Equivalence

Consider a hypothetical trial with an estimated risk ratio of 1.02 and a 95% confidence interval from 0.94 to 1.11. Suppose the clinically negligible range was set in advance as 0.90 to 1.10.

Because the interval contains 1, a conventional two-sided test might fail to reject no difference. But the upper limit, 1.11, is outside the equivalence margin’s upper limit of 1.10. The interval is therefore not wholly within the prespecified range, so this result does not establish equivalence under the stated criterion.

The distinction is about the question being answered. “We did not detect a difference” describes the outcome of a conventional test. “The effects are equivalent within specified bounds” requires a sufficiently precise estimate contained within clinically justified bounds.

The published RECOVERY interval, 0.97 to 1.23, likewise cannot be read as proof of equivalence. It includes 1 but also extends to a mortality rate ratio of 1.23, which is compatible with a material increase in the mortality rate. It does not include a large mortality-rate reduction such as a mortality rate ratio of 0.90, but whether a particular remaining range counts as clinically meaningful must be judged against a margin set for that purpose, not chosen after seeing the result.

How to Read the Decision and Its Limits

A recruitment stop for futility protects participants and trial resources when continued assignment is unlikely to answer the treatment question usefully. It also has costs: stopping reduces the information that could have accrued, and a decision based on an interim estimate is inherently less precise than a final analysis. A monitoring committee must weigh the prospect of useful information against those costs and the ethical implications of continuing.

The final RECOVERY result is consistent with the decision not to continue hydroxychloroquine recruitment: the point estimate did not suggest a mortality benefit, and the reported interval did not support a large reduction in 28-day mortality. However, the interval’s upper end remains compatible with increased mortality, and uncertainty remains about effects smaller than the trial could rule out. The appropriate conclusion is not “the treatments were identical,” but that the evidence did not support continuing recruitment to seek a meaningful benefit in this setting.

The scope matters. The decision concerns the hydroxychloroquine comparison and the population and outcome studied: hospitalized patients and 28-day mortality. It does not establish effects in every disease stage, on every outcome, or under every dosing regimen. Nor does stopping one comparison mean that the wider platform trial stopped evaluating other treatments.

Diagnostics and Common Pitfalls

Confusing a stopping decision with a final estimate. Interim evidence informs the decision at the time; the final published counts may include additional follow-up or participants. Do not present final counts as the committee’s exact interim dataset.
Treating \(p>0.05\) as evidence of equivalence. A nonsignificant test may reflect imprecision. Equivalence needs a prespecified margin and an appropriate interval-based or two-one-sided-test procedure.
Calling futility proof of no effect. Futility concerns the chance or value of achieving a trial objective, conditional on assumptions. It does not establish that the true effect is exactly zero.
Ignoring the assumed future effect in conditional power. Conditional power can differ substantially under a design alternative and under continuation of the interim trend. Report the assumption and examine sensitivity to plausible alternatives.
Overinterpreting the simple risk-ratio calculation. The count-based Wald interval is approximate and unadjusted. Use the trial’s prespecified analysis for its primary inference, and label a simpler calculation as illustrative.

Key takeaway: RECOVERY stopped hydroxychloroquine recruitment after interim evidence indicated no beneficial effect worth pursuing. That is a futility judgment about continuing recruitment, not proof of equivalence. The final mortality estimate and its uncertainty should be interpreted in the population, outcome, and analysis to which they apply.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

In a hypothetical trial, conditional power is 12% under the prespecified design alternative and the protocol’s nonbinding futility threshold is 15%. What does this imply, and what does it not prove?
Why does a 95% confidence interval that includes a risk ratio of 1 fail to establish equivalence?
Using the final RECOVERY counts, why is it inaccurate to say that hydroxychloroquine reduced mortality because the approximate interval includes values below 1?
What is one reason the approximate risk-ratio interval calculated from the counts may differ from the published RECOVERY interval?

Solutions. (1) Under the specified assumptions, the estimated chance of meeting the final success criterion is below the protocol’s threshold, supporting a futility recommendation if other considerations agree. It does not prove a zero treatment effect or rule out all clinically relevant effects. (2) Containing 1 only means no difference is among the effects compatible with the interval; equivalence requires the entire suitable interval to lie within prespecified clinical margins. (3) Values below 1 are compatible with the data, but so are 1 and values above 1; compatibility is not evidence that mortality was reduced. The point estimate is above 1. (4) The hand calculation uses a simple unadjusted large-sample formula, while the published estimate comes from the trial’s analysis and may use different methods or estimands.