Why the Effect Scale Matters at an Interim Look
In a multi-arm trial with a binary endpoint, each participant either experiences the event defined as a response or does not. At an interim analysis, the observed response counts provide simple scores for comparing experimental arms with a shared control. These scores can be expressed as response rates, risk differences, or risk ratios. The calculations are straightforward, but their scales answer different questions: an absolute difference describes percentage-point gain, while a ratio describes relative gain.
An arm-selection rule must specify how these scores feed into the decision. For example, a rule might retain the top-ranked experimental arms, or retain arms that exceed a prespecified clinical threshold. For a common control and unadjusted point estimates, the three scores rank arms identically when the observed control response rate is positive. Their numerical values and clinical interpretations are not interchangeable, however, and a risk ratio may not be defined when no control participants respond.
For the basic calculations in this tutorial, assume that participants are randomized to the arms, each participant contributes one binary outcome, and outcomes are assessed for everyone included in the interim denominators. Treat the observed counts in different arms as independent binomial counts: \(X_j\sim\operatorname{Binomial}(n_j,p_j)\), where \(p_j\) is the arm's response probability. The control is concurrent and common to the comparisons, and a response has the same definition in every arm. These assumptions make the estimates interpretable as comparisons of response probabilities; they do not by themselves guarantee that a selection rule has a particular error rate or operating characteristic.
If the event is unfavorable, such as disease progression, higher event rates do not favor treatment. The outcome or score direction must then be defined consistently so that the selection rule favors the clinically desirable result. The examples below use a favorable response, with higher response rates preferred.
How the Three Scores Relate
The scores share the same observed arm rates. Response-rate scoring asks how often participants responded in the experimental arm. Risk-difference scoring subtracts the control rate, expressing the estimated absolute gain. Risk-ratio scoring divides by the control rate, expressing the estimated relative response.
For two experimental arms \(a\) and \(b\), a common control rate gives
The first equivalence follows by subtracting the same quantity, \(\hat p_0\), from both arm rates. For the second, divide both sides of \(\hat p_a>\hat p_b\) by the same positive quantity, \(\hat p_0\); division by a positive number preserves the inequality. Therefore, if the rule simply ranks arms by these point estimates and retains the top \(K\), all three scores select the same arms, apart from ties. This fact does not mean the scales have the same interpretation or that every threshold-based rule gives the same decision.
Worked Example: Calculating Scores and Ranking Three Arms
Consider hypothetical interim data from a trial with one control and two experimental arms. All participants in the interim denominators have an assessed outcome, and response is favorable.
| Arm | Responders | Assessed | Response rate |
|---|---|---|---|
| Control | 12 | 40 | 0.300 |
| A | 20 | 40 | 0.500 |
| B | 16 | 40 | 0.400 |
The observed rates are \(\hat p_0=12/40=0.300\), \(\hat p_A=20/40=0.500\), and \(\hat p_B=16/40=0.400\). Thus,
and
The response-rate scores are 0.500 for A and 0.400 for B. All three scoring methods rank A above B. The estimates describe an observed 20-percentage-point response-rate difference and a response rate 1.667 times the control rate for A; they do not establish that the true effects have those exact values.
Thresholds Must Match Their Scale
A threshold rule adds a separate policy choice to the score calculation. For a fixed observed control rate, thresholds can be translated exactly between scales. If the minimum acceptable absolute gain is \(\delta\), the risk-difference rule retains an arm when \(\hat p_k-\hat p_0\geq\delta\). This is equivalent to requiring \(\hat p_k\geq\hat p_0+\delta\). If the minimum acceptable relative response is \(\rho\), the risk-ratio rule requires \(\hat p_k\geq\rho\hat p_0\), provided \(\hat p_0>0\). A response-rate threshold \(c\) directly requires \(\hat p_k\geq c\).
These translations show why the same numerical cutoff should not be copied across scales. A cutoff of 0.10 for a risk difference means a gain of 10 percentage points; a cutoff of 0.10 for a risk ratio means one-tenth of the control response rate. The clinical meaning differs. Also, when the observed control rate varies, a fixed relative threshold corresponds to a different absolute gain, and a fixed absolute threshold corresponds to a different relative gain.
Worked Example: Translating a Clinical Retention Threshold
Use the hypothetical data in the previous example. Suppose the interim rule requires at least a 15-percentage-point gain over control. For arm A, the estimated difference is \(0.500-0.300=0.200\), so A exceeds the threshold. For B, it is \(0.400-0.300=0.100\), so B does not.
The equivalent response-rate cutoff is \(0.300+0.150=0.450\). A's rate of 0.500 meets it, while B's rate of 0.400 does not. The equivalent risk-ratio cutoff is \(1+0.150/0.300=1.500\), because a 0.150 absolute increase over a 0.300 control rate corresponds to a ratio of \(0.450/0.300\). A's ratio of 1.667 meets this criterion and B's ratio of 1.333 does not. All three criteria retain A and do not retain B because the thresholds describe the same observed response-rate requirement.
Now consider three different policy cutoffs: response rate at least 0.400, risk difference at least 0.150, and risk ratio at least 1.500. The response-rate criterion retains both A and B; the other two retain only A. This is not a contradiction: the three cutoffs encode different requirements. The response-rate criterion asks only for a rate of at least 0.400, whereas the other criteria demand a larger improvement over the observed control rate.
What the Scales Emphasize
| Score | Interpretation | Useful feature | Important limitation |
|---|---|---|---|
| Response rate, \(\hat p_k\) | Observed proportion responding in the experimental arm | Directly communicates the arm's observed outcome frequency | Does not by itself express improvement over control |
| Risk difference, \(\widehat{RD}_k\) | Observed absolute gain or loss in response probability | Expresses a clinically interpretable difference in percentage points | The same difference can represent different relative gains at different control rates |
| Risk ratio, \(\widehat{RR}_k\) | Observed experimental response rate divided by the control rate | Expresses relative response and is unchanged by a common multiplicative rescaling of rates | Undefined when the observed control rate is zero; relative effects can obscure absolute gains |
With a common control, top-\(K\) selection by these unadjusted scores has the same ranking whenever the control rate is positive. A practical alternative is to rank by a confidence bound or another uncertainty-adjusted score. Such a score can account for precision, but its construction and decision rule must be specified separately; it is not the same as ranking the three raw point estimates. In particular, different interim sample sizes do not alter the algebraic equivalence of the raw rankings, but they can make the estimates differ substantially in precision.
Worked Example: Same Absolute Gain, Different Relative Gain
Compare two hypothetical settings, each with 100 assessed participants in control and treatment. In setting 1, 10 control participants and 20 treated participants respond. In setting 2, 50 control participants and 60 treated participants respond.
In setting 1, the rates are \(0.10\) and \(0.20\). The estimated difference is \(0.20-0.10=0.10\), or 10 percentage points, and the estimated ratio is \(0.20/0.10=2.00\). In setting 2, the rates are \(0.50\) and \(0.60\). The estimated difference is \(0.60-0.50=0.10\), also 10 percentage points, while the estimated ratio is \(0.60/0.50=1.20\).
The absolute gains are equal, but the relative gains are not. Thus a decision-maker using a risk-difference threshold treats the observed gains equally, while a decision-maker using a risk-ratio threshold distinguishes them. Neither scale is universally preferable: the relevant scale depends on the clinical question and on a prespecified retention policy.
Zero Control Responses and Other Failure Modes
When \(\hat p_0=0\), division by the observed control rate is impossible. A risk ratio based on the raw observed rates is therefore undefined, not zero and not infinite. The response rates and risk differences remain calculable, although a small interim sample with no control responses can make the apparent differences unstable. Adding an arbitrary constant to the counts to force a finite ratio changes the estimator and can affect selection. Such a correction should not be introduced ad hoc; any alternative estimator or rule needs to be specified and justified in advance.
Worked Example: A Zero Control Rate
Suppose a hypothetical interim analysis has 0 responses among 20 control participants, 3 among 20 participants in arm A, and 1 among 20 in arm B. The observed rates are \(\hat p_0=0/20=0\), \(\hat p_A=3/20=0.150\), and \(\hat p_B=1/20=0.050\).
The estimated differences are \(\widehat{RD}_A=0.150-0=0.150\) and \(\widehat{RD}_B=0.050-0=0.050\). The response-rate scores likewise rank A above B. But the raw risk ratios would require dividing \(0.150\) and \(0.050\) by zero, so neither is defined. The data support calculating these observed rates and differences, not claiming that either arm has a finite observed risk ratio. With only 20 participants per arm, the zero control count also signals that a selection decision based solely on these point estimates may be sensitive to individual outcomes.
Other checks help prevent avoidable errors. Confirm that each denominator is the number of participants actually represented by the response count and that response has a consistent definition across arms. Verify that the common control is the intended comparator for each experimental arm. If the observed control rate is positive but very small, the risk ratio is defined but may be numerically large and sensitive to a small change in counts. Finally, a favorable point estimate is not proof of a clinically useful effect: an interim selection score is a decision input, not a certainty statement about the true response probabilities.
Applying a Binary Score Carefully
Confirm that response is the favorable event, counts are valid, and the interim denominators refer to assessed participants under the specified analysis.
For each arm, compute \(\hat p_j=x_j/n_j\). Keep the observed rates available even if the rule will use differences or ratios.
Calculate the risk difference or, only when \(\hat p_0>0\), the raw risk ratio. State the scale and units clearly.
Use the designated ranking or clinically meaningful cutoff. Translate thresholds before comparing scales; do not substitute a cutoff with a different meaning.
The central practical distinction is between ranking and qualification. If the control rate is positive, raw rates, differences, and ratios give the same ranking under a shared control. But qualification against a threshold depends on the threshold's scale and clinical interpretation. If the control rate is zero, rate and difference scores can still be computed while the raw risk ratio cannot. The choice of score should therefore be made before observing the interim results and aligned with the intended decision.
Check Your Understanding
Answer each question using the definitions and rules above. Full solutions follow the questions.
- A hypothetical control arm has 18 responses among 60 participants. Experimental arms A and B have 30 and 24 responses among 60 participants each. Calculate their response rates, risk differences, and risk ratios, and state the raw-score ranking.
- With an observed control response rate of 0.25, what response-rate cutoff is equivalent to a risk-difference threshold of 0.10? What risk-ratio threshold represents that same criterion?
- Why is a raw risk ratio undefined when there are no observed control responses? Is it correct to report the ratio as zero?
- Two experimental arms have observed rates 0.42 and 0.36, and they share a positive control rate of 0.30. Which arm ranks higher by response rate, risk difference, and risk ratio? Explain.
Solutions. (1) The control rate is \(18/60=0.300\). For A, \(\hat p_A=30/60=0.500\), \(\widehat{RD}_A=0.500-0.300=0.200\), and \(\widehat{RR}_A=0.500/0.300=1.667\). For B, \(\hat p_B=24/60=0.400\), \(\widehat{RD}_B=0.400-0.300=0.100\), and \(\widehat{RR}_B=0.400/0.300=1.333\). A ranks above B by all three raw scores. (2) The equivalent response-rate cutoff is \(0.25+0.10=0.35\). The equivalent ratio cutoff is \(0.35/0.25=1.40\). Each criterion requires the experimental response rate to be at least 0.35. (3) The raw ratio requires division by the observed control rate, so a zero denominator makes it undefined. It is not correct to report zero; that would be a defined value with a different meaning. (4) The differences are \(0.42-0.30=0.12\) and \(0.36-0.30=0.06\); the ratios are \(0.42/0.30=1.40\) and \(0.36/0.30=1.20\). The arm with rate 0.42 ranks higher on all three scores because the control rate is shared and positive.