Why the Outcome Scale Matters
A response-adaptive rule uses accumulated outcomes to influence later assignments. But “the outcome” is not always a single, self-evident quantity. The same patients might be described by whether they met a responder threshold, by the number of symptom-free days they experienced, or by a utility score that also accounts for adverse effects. These are different statistical representations, and they generally target different treatment contrasts.
Consequently, changing the outcome scale can change more than the numerical units. It can change which clinical differences count as important, how much information the model extracts, and how uncertain the estimated treatment contrast is. An allocation rule then translates that model-based evidence into assignment probabilities. It is essential to distinguish the influence of the outcome model from the influence of that final translation rule.
The comparisons below use a hypothetical two-arm trial. Each patient has a binary responder status, symptom-free days, and a toxicity indicator. For illustration, response is defined as at least five symptom-free days. Utility is defined in advance as symptom-free days minus three points for experiencing toxicity. Thus the binary outcome discards the amount of benefit beyond the threshold; the continuous outcome retains that amount; and the utility outcome trades symptom relief against toxicity using an explicit clinical weight.
Model and Allocation Rule for the Comparison
To make the scale comparison concrete, suppose 20 hypothetical patients have been assigned to each arm. Arm A has 14 responders out of 20 and Arm B has 11 out of 20. The mean symptom-free days are 6.2 in A and 5.5 in B, with sample standard deviations 2.0 and 2.5 days. Toxicity occurs in 6 of 20 patients in A and 2 of 20 in B. The corresponding mean utilities are 5.3 and 5.2 utility points, with sample standard deviations 2.4 and 2.2 points. These summaries are mutually compatible: for example, the mean utility in A is \(6.2-3(6/20)=5.3\).
For a transparent comparison, let \(\pi_A\) denote the model-based probability that A has a better value than B on the chosen scale. Use the illustrative allocation rule
where \(\kappa>0\) controls how strongly the rule responds to evidence. The quantity \(2\pi_A-1\) is a signed, dimensionless measure: it is positive when A is more likely to be better, negative when B is more likely to be better, and zero when the model assigns equal probabilities. Here set \(\kappa=0.5\). This is one illustrative rule, not a universal prescription; other adaptive designs use different scores and mappings.
Worked Example: Binary Responder Scale
Model each arm's response probability independently with a Beta(1,1) prior. With 14 responders and 6 nonresponders in A, and 11 responders and 9 nonresponders in B, the posterior distributions are \(p_A\mid data\sim\mathrm{Beta}(15,7)\) and \(p_B\mid data\sim\mathrm{Beta}(12,10)\). Their posterior means are \(15/22=0.6818\) and \(12/22=0.5455\); the estimated difference is \(0.1364\).
For a hand calculation, approximate each posterior by a normal distribution with its exact Beta mean and variance. For a Beta(\(a,b\)) distribution, the variance is \(ab/[(a+b)^2(a+b+1)]\). Thus the posterior variance of the difference, using independent arm posteriors, is approximately
The posterior standard deviation of the difference is \(\sqrt{0.02021}=0.1422\). Therefore the normal approximation gives
The allocation probability is then \(q_A=1/[1+\exp\{-(2(0.831)-1)/0.5\}]\). The exponent's positive argument is \(0.662/0.5=1.324\), so \(q_A\approx0.790\); \(q_B\approx0.210\). Under this model and rule, the next patient is assigned to A with probability about 79.0%. The calculation depends on the responder threshold and the Beta prior, and the normal approximation is only a convenient approximation to the posterior comparison.
Worked Example: Continuous Symptom-Free Days
Now retain each patient's symptom-free days rather than reducing them to responder status. Assume independent, approximately normal outcomes within arms, and use a weak, diffuse prior for each arm's mean. With the sample summaries above, the estimated mean difference is \(6.2-5.5=0.7\) days. Using the sample standard deviations as plug-in estimates, its standard error is
A normal approximation to the posterior for the mean difference gives
Using the same allocation rule and \(\kappa=0.5\), \(q_A=1/[1+\exp\{-(2(0.836)-1)/0.5\}]\). Since \(2(0.836)-1=0.672\), this is \(\operatorname{logistic}(1.344)\approx0.793\), or 79.3%. A and B therefore receive similar allocation probabilities under these particular data and assumptions, even though the responder and continuous analyses estimate different contrasts.
The calculation assumes the sample means and variances are informative for the outcome distribution and that observations are independent within and between arms. Strong skewness, influential outliers, or dependence between patients would undermine this simple model. A small sample also makes the plug-in normal approximation uncertain.
Worked Example: Utility Scale Including Toxicity
Define each patient's utility as symptom-free days minus three points if toxicity occurs. The weight of three is a hypothetical clinical choice, not something inferred from these data. With mean utilities 5.3 and 5.2, A's estimated advantage is only \(0.1\) utility points. Under the same approximate normal model, the standard error is
Thus \(\pi_A\approx\Phi(0.1/0.7280)=\Phi(0.137)\approx0.555\). Applying the same dimensionless allocation mapping gives \(q_A=1/[1+\exp\{-(2(0.555)-1)/0.5\}]\). The argument of the logistic function is \(0.110/0.5=0.220\), so \(q_A\approx0.555\), with \(q_B\approx0.445\). The relatively balanced assignment reflects weak evidence of a utility advantage, despite the more favorable A estimates on the other two scales.
This contrast is clinically meaningful only if the utility definition is defensible. A different toxicity penalty could produce a different mean contrast and allocation probability. The rule cannot decide whether three points is an appropriate trade-off; that choice must be justified before using outcomes to adapt assignment.
What the Comparison Shows
The calculations use the same hypothetical patients, but each scale answers a different question. The binary model compares response probabilities at a particular cutoff. The continuous model compares expected symptom-free days. The utility model compares expected values under an explicit weighting of benefit and harm. These estimands need not rank treatments identically, and the amount of uncertainty can differ because dichotomization discards information while a continuous model must account for the outcome's distribution and variability.
The allocation rule matters too. In the three examples, the same mapping from posterior probability to assignment probability gave approximately 79.0%, 79.3%, and 55.5% assignment to A. These are outputs of both the scale-specific model and the chosen mapping; they are not intrinsic properties of the treatments. In particular, the probability of assignment is not itself a probability that treatment A is effective or safe.
Another common rule uses posterior expected rewards directly. For example, if the rewards are on a common, prespecified utility scale, a two-arm softmax rule can be written as
where \(\bar U_A\) and \(\bar U_B\) are posterior expected utilities and \(\tau\) has the same units as utility. This rule is different from the posterior-superiority rule above. Its temperature parameter must be interpreted on the utility scale; changing utility units without changing \(\tau\) changes the allocation.
Worked Example: Checking a Softmax Allocation Calculation
Suppose a separate hypothetical calculation gives posterior expected rewards 0.650 and 0.477 on a common utility scale, with \(\tau=0.2001\) utility units. Then
The difference is \(0.173\), and \(0.173/0.2001\approx0.86457\). The resulting allocation probability is about 70.4%, not 70.3%. If rewards were multiplied by a constant, \(\tau\) would also need to be multiplied by that constant to preserve this allocation probability.
Assumptions, Diagnostics, and Limitations
A scale comparison is credible only when the outcome definitions and models fit the clinical question. Check that the responder cutoff is clinically motivated, the utility formula and weights were specified independently of the observed treatment differences, and the quantitative model is plausible for the observed outcome distribution. For a binary model, examine whether the prior has a noticeable influence when event counts are small. For a continuous model, inspect the spread and shape of outcomes, and consider whether arm-specific variances are plausible. For utility, inspect the component outcomes as well as the composite: a single mean can conceal a clinically concerning trade-off.
These checks do not establish that adaptive allocation is statistically valid for every purpose. They assess whether the scale-specific input is sensible. Allocation can become erratic when the model is highly uncertain or driven by a few early observations; it can also consistently favor a treatment for a poorly chosen endpoint. A good allocation probability does not guarantee good estimation or valid hypothesis testing. Those operating characteristics must be assessed for the full design and analysis, rather than inferred from a single allocation calculation.
The main alternatives are not interchangeable. A binary endpoint is easy to interpret and may directly match a clinical decision, but a threshold loses distinctions among patients on the same side of the cutoff. A continuous endpoint preserves gradations and can be more informative when its model is adequate, but its mean may not capture important harms. A utility endpoint combines outcomes into one decision-relevant measure, but its weighting assumptions must be transparent and acceptable. A careful design may report several outcomes, while specifying which one drives adaptation; otherwise the allocation target is ambiguous.
Decide whether the goal concerns a response threshold, an average quantitative outcome, or an explicit benefit-harm trade-off.
State the outcome definition, assumptions, and uncertainty calculation before translating evidence into assignments.
Confirm the score, tuning parameter, and units; then verify that stronger evidence for an arm changes assignment in the intended direction.
Common Mistake
A common mistake is to say, “The continuous scale gave A a larger numerical advantage, so the adaptive rule should assign more patients to A.” Raw differences cannot be compared across scales: 0.7 days, 0.136 responder-probability units, and 0.1 utility points have different meanings and units. Even within one scale, a difference alone does not describe uncertainty. The correction is to state the estimand and outcome model, quantify uncertainty on that scale, and apply a clearly specified allocation rule whose parameters are calibrated to its score. If the clinical weighting or threshold is debatable, show how conclusions change under plausible alternatives rather than treating one representation as uniquely correct.
Check Your Understanding
Answer each question using the definitions and calculations in this tutorial.
- If a responder cutoff is changed from five symptom-free days to seven, what part of the analysis changes, and why is this not merely a change of units?
- For the continuous example, calculate the standardized mean difference used in the normal approximation and explain its role.
- If a toxicity penalty is increased while all observed outcomes stay fixed, can the utility-based allocation probability change? Explain.
- Why must \(\tau\) change if utility scores and their differences are multiplied by ten in the softmax rule?
Solutions. (1) The binary outcome definition and hence the responder counts, estimated response contrast, and allocation input change; patients can switch categories, so the estimand changes. (2) The standardized difference is \(0.7/0.7159=0.978\), which is the estimated mean contrast measured in its standard-error units and is used to approximate the posterior probability that A's mean is larger. (3) Yes. A larger penalty lowers utility more for the arm with more toxicity, changing the utility contrast and potentially its uncertainty and allocation probability. (4) The ratio of the expected-utility difference to \(\tau\) determines the softmax probability. Multiplying both by ten preserves that ratio; multiplying the rewards alone does not.