Why Count More Than Successes?
A success count treats each patient as either a success or a non-success. That can be useful when a threshold has clear clinical meaning, but it discards information about how much patients improved. If symptom improvement is measured on a continuous scale, an allocation rule that gives more patients a better-performing treatment might change the average improvement, the number reaching an important threshold, or both. Those are distinct summaries, and they need not lead to the same conclusion.
The question here is not whether a treatment works in general. It is how to summarize patient outcomes under different allocation policies: for example, a fixed rule that assigns half of patients to each treatment, or a response-adaptive rule that changes assignment probabilities as trial data accumulate. A policy-level comparison describes outcomes for patients enrolled under each rule. It does not, by itself, establish that the allocation policy caused a particular individual to benefit.
If the outcome is improvement, one possible measurement is baseline symptom score minus follow-up symptom score, so that positive values represent improvement. The scale, follow-up time, and direction of benefit must be specified before comparing policies. A difference of 1.2 points has no useful interpretation until the outcome scale and units are known.
Deriving the Policy-Level Mean
Let \(H_i\) denote the trial history available just before patient \(i\) is assigned, and let \(p_i(g)=P(A\mid H_i)\) be the probability that policy \(g\) assigns patient \(i\) to A. The corresponding probability of B is \(1-p_i(g)\). The assignment must be randomized according to these probabilities, rather than determined by a patient's unobserved potential outcomes. Assume consistency (the observed outcome equals the potential outcome under the treatment assigned), no interference between patients, and that the outcome is measured comparably under both treatments. Conditional means may depend on the history and patient characteristics; write \(\mu_{Ai}=E[Y_i(A)\mid H_i]\) and \(\mu_{Bi}=E[Y_i(B)\mid H_i]\).
The expected outcome for patient \(i\), conditional on the available history, is a weighted average of the two conditional treatment means. This follows by separating the two mutually exclusive assignment events:
Taking expectations over the histories generated under policy \(g\), then averaging across patients, gives
The history matters: under response-adaptive randomization, later assignment probabilities depend on earlier outcomes, and the distribution of those histories is itself generated by the policy. Thus, plugging a final allocation fraction into a formula can be a useful conditional calculation, but it is not generally a complete calculation of the policy's expected benefit. To evaluate a policy expectation, one must account for the possible histories and allocation probabilities it produces.
Worked Comparisons
Worked Example: Average Improvement Under Two Allocation Ratios
Consider a hypothetical trial of 40 patients using symptom improvement in points at a fixed follow-up. For an illustrative calculation, suppose the relevant mean improvement is 6.0 points on A and 4.0 points on B, and treat those means as known for this calculation. Compare a fixed 1:1 allocation with a particular allocation that assigns 30 patients to A and 10 to B.
Under 1:1 allocation, the expected total improvement using these stipulated means is
The average is \(200/40=5.00\) points per patient. Under the 30:10 allocation, the corresponding total is
In this stipulated allocation scenario, the difference is 20 total points, or \(0.50\) points per enrolled patient, in favor of the 30:10 allocation. This arithmetic does not show that a response-adaptive rule will achieve that gain: it assumes the displayed treatment means apply and that the allocation is as specified. A rule that must learn which treatment has the higher mean can allocate differently across possible trial histories.
Worked Example: A Clinically Important Improvement Threshold
Suppose a hypothetical symptom scale records improvement in points, and an improvement of at least 5 points is defined in advance as clinically important. Under one allocation scenario, 15 patients receive A and 5 receive B. Suppose A has a mean improvement of 5.0 points and an estimated 60% of its patients reach the threshold; B has a mean improvement of 4.0 points and an estimated 25% reach it.
The average improvement across the 20 patients, using these stipulated arm summaries, is
The expected number reaching the threshold is \(15(0.60)+5(0.25)=9+1.25=10.25\), or \(10.25/20=51.25\%\) of patients. The fractional count is an expected count based on the stipulated rates; an actual trial count must be an integer.
For comparison, a 10:10 allocation with the same arm summaries has average improvement \(\{10(5.0)+10(4.0)\}/20=4.50\) points and expected threshold count \(10(0.60)+10(0.25)=8.5\), or 42.5%. In this scenario, both summaries favor the 15:5 allocation. The threshold summary, however, answers a different question from average improvement: it reports how many patients are expected to reach the chosen cutoff, not how far patients improve overall.
Worked Example: Equal Means, Different Threshold Benefit
Now consider a different hypothetical situation. A patient assigned to A has a 50% chance of improving by 10 points and a 50% chance of improving by 0 points. A patient assigned to B improves by 5 points. Both treatments therefore have a mean improvement of 5 points. Define clinically important improvement as at least 6 points.
Compare 10:10 allocation with an allocation of 16 patients to A and 4 to B, for 20 patients total. Under either allocation, expected total improvement is \(20(5)=100\) points, so expected average improvement is \(100/20=5.00\) points per patient.
Under 10:10 allocation, each A patient has probability 0.50 of reaching the threshold and no B patient reaches it. The expected threshold count is \(10(0.50)+10(0)=5\), or 25% of patients. Under 16:4 allocation it is \(16(0.50)+4(0)=8\), or 40%. Thus the mean improvement is unchanged, while the expected number reaching this particular threshold is higher with more assignments to A. This difference reflects the outcome distributions and the chosen cutoff, not a difference in the treatments' mean improvement.
Choosing and Interpreting a Benefit Summary
The mean outcome uses the full measured scale and is often a direct summary when differences in points have consistent meaning. It can conceal important features of the outcome distribution, such as a mixture of large improvements and no improvement. A threshold-based summary can communicate the proportion expected to reach a clinically important level, but its meaning depends on the cutoff. Patients just below and just above the threshold may have very similar outcomes while being classified differently.
A treatment-success count is therefore not a universal substitute for a continuous outcome. If a threshold is clinically established and relevant, report the threshold-based summary alongside the continuous one rather than assuming one explains the other. Also state whether a reported number is an observed count, an estimated probability, or an expected count under specified assumptions.
| Summary | Question answered | Key interpretive caution |
|---|---|---|
| Expected average improvement | What is the average improvement per enrolled patient under this policy? | May hide variability or distinct outcome subgroups. |
| Expected total improvement | What is the combined improvement across the trial's patients? | Depends on the number enrolled; do not compare totals across different sample sizes without context. |
| Expected proportion above a threshold | What proportion is expected to reach a specified improvement level? | Depends on the threshold and discards distances from it. |
| Arm-specific outcome summaries | What outcomes are summarized among patients assigned to each treatment? | Do not confuse a descriptive arm summary with the policy-level outcome for all enrolled patients. |
These summaries answer a different question from an average treatment-effect estimate. The latter compares potential outcomes under A and B for a defined population; a policy-level mean describes outcomes generated by a particular allocation rule. A policy can change the mix of treatments patients receive without changing either treatment's underlying outcome distribution. State which quantity is being compared before interpreting a numerical difference.
Assumptions, Diagnostics, and Limitations
A meaningful comparison requires the same outcome definition, units, measurement schedule, and target patient population for the policies being compared. The allocation probabilities must be implemented as specified, and outcomes should be observed comparably across treatment groups. If outcome data are missing, the summary needs an explicit approach to that missingness; treating unobserved outcomes as if they were known can distort either the mean or the threshold proportion. The no-interference assumption is also important: one patient's assignment should not change another patient's outcome.
When reviewing a comparison, check that the continuous scale is oriented consistently so that larger values really do mean better outcomes. Verify the threshold and its clinical interpretation before calculating a threshold proportion. Examine the arm-specific outcome distributions as well as their means, and report the policy's allocation probabilities or allocation counts. For a response-adaptive policy, inspect how probabilities changed over time and whether the comparison represents the full rule across possible histories or only one realized allocation sequence.
A realized trial's final treatment counts and observed arm means can describe that trial, but they do not alone establish the expected benefit of the allocation policy. The policy's probabilities depend on earlier data, so its expectation involves possible histories, not just the final fraction assigned to each treatment. Further, a positive difference in expected average improvement is not a guarantee of improvement for every patient; it summarizes outcomes over the specified population and assumptions.
Common Mistake
Wrong reasoning: “The adaptive allocation assigned more patients to the treatment with the higher observed mean, so it necessarily gave patients more benefit.” This treats an observed mean and one realized allocation as if they were the true treatment means and the full policy expectation. It also leaves “benefit” undefined.
Correction: Specify the patient outcome and summary first, then compare the expected summary under each allocation policy for the same population and sample size. For response-adaptive allocation, the calculation or evaluation must account for the histories that determine future probabilities. Report whether the result concerns average points, total points, a threshold proportion, or another clearly defined summary.
Why the Summary Matters
Comparing allocation rules is not complete until the patient outcome and the benefit summary are explicit. Average improvement emphasizes the scale as a whole; total improvement additionally reflects the number of patients; and threshold attainment emphasizes a particular clinical cutoff. As the examples show, these summaries can agree, differ in magnitude, or tell different stories. None is automatically the right choice for every setting. The appropriate summary depends on the clinical question, the outcome scale, and the interpretation intended for patients and decision-makers.
Check Your Understanding
Answer each question using the outcome definition and assumptions stated in the problem.
- In a trial of 30 patients, fixed allocation assigns 15 to A and 15 to B. Their stipulated mean improvements are 7 and 5 points. What are the expected total and average improvements?
- Why can the expected total improvement be a poor comparison if two policies enroll different numbers of patients?
- Two treatments have the same mean improvement, but different proportions of patients exceed a clinically important threshold. Is this a contradiction? Explain.
- A response-adaptive rule changes assignment probabilities based on earlier outcomes. Why might the final allocation fraction alone be insufficient to calculate its expected policy-level mean?
Solutions. (1) The expected total is \(15(7)+15(5)=105+75=180\) points; the expected average is \(180/30=6.00\) points per patient. (2) A larger total may result simply from enrolling more patients, so totals should be compared at the same sample size or interpreted with the number enrolled clearly stated. (3) No. The mean summarizes the full scale, whereas the threshold proportion depends on how outcomes are distributed around the cutoff. (4) The allocation probabilities depend on earlier outcomes, and those histories can vary. The policy expectation must account for the histories and resulting assignments, not merely one final fraction.