Why Profile-Specific Probabilities Need Careful Interpretation
In a covariate-adjusted adaptive design, the chance of assigning a patient to a treatment may depend on the patient’s baseline profile and on information accumulated during the trial. A probability reported for one profile is therefore a description of the assignment mechanism at a particular point in the trial—not, by itself, a measure of how much that patient will benefit from treatment.
The distinction matters because profiles can differ in prognosis: patients with one profile may have better expected outcomes under either treatment. They may also differ in treatment effect: the contrast between treatments may genuinely vary by profile. These are different claims. To interpret an allocation probability, first identify what probability it represents, then compare it with the profile-specific treatment contrasts implied by the outcome model and the allocation rule.
With two treatments, A and B, the probabilities satisfy \(\pi_i(A\mid x,h)+\pi_i(B\mid x,h)=1\). A profile comparison is most interpretable when the same trial history and allocation rule are held fixed. If the probability is updated after each patient, values recorded at different times may differ because the history changed, even when the profile is identical.
Prognosis Is Not the Same as Treatment Effect
Let \(Y\) be an outcome for which larger values are better, and code treatment as \(A=0\) for A and \(A=1\) for B. A useful mean model for interpretation is
Here \(m(x)\) is the expected outcome under treatment A for profile \(x\), and \(\tau(x)\) is the expected difference between B and A for that profile. The function \(m(x)\) describes prognosis under the reference treatment; \(\tau(x)\) describes the conditional treatment contrast. This representation concerns conditional means and does not require a particular outcome distribution.
The distinction follows directly by subtracting the two treatment-specific means. For profile \(x\), the expected outcome under B is \(m(x)+\tau(x)\); under A it is \(m(x)\). Thus the treatment contrast is
The prognostic term cancels. If \(\tau(x)=\tau\) for every profile, expected outcomes may still vary substantially across profiles through \(m(x)\), but the model has no treatment-effect heterogeneity on this mean-difference scale. Conversely, evidence of heterogeneity requires comparing treatment contrasts across profiles, not merely comparing their outcome levels or allocation probabilities.
An allocation algorithm may use estimated response information, exploration, and design constraints to calculate probabilities. Therefore, a profile-specific probability can vary for several reasons. It may reflect a treatment contrast estimated differently by profile; it may reflect other elements of the allocation rule; or the compared probabilities may have been calculated under different histories. A table of probabilities alone does not identify which explanation applies. The rule and the conditions used to calculate the probabilities must be stated.
Worked Example: Reading a Profile-Specific Probability Table
Suppose a hypothetical two-treatment trial reports probabilities for assigning the next patient to B. The values are calculated using the same recorded trial history and the same version of the allocation rule.
| Patient profile | Estimated probability of B | Estimated probability of A |
|---|---|---|
| Lower baseline-risk profile | 0.56 | 0.44 |
| Higher baseline-risk profile | 0.68 | 0.32 |
For the lower-risk profile, the estimated chance of B is 0.56, or 56%; for the higher-risk profile, it is 0.68, or 68%. The difference is \(0.68-0.56=0.12\), or 12 percentage points. The corresponding probabilities of A are the complements: \(1-0.56=0.44\) and \(1-0.68=0.32\).
The correct interpretation is that, under this rule and the specified common history, the next patient with the higher-risk profile has a 12-percentage-point larger conditional chance of assignment to B. The table does not show that B is more effective for higher-risk patients. To support that conclusion, one would need to compare the model’s estimated treatment contrasts across the profiles and understand how those estimates enter the allocation rule.
Use Treatment Contrasts to Assess Heterogeneity
A direct way to distinguish prognosis from treatment-effect differences is to calculate each profile’s predicted outcome under each treatment, then subtract within profile. The following example uses a hypothetical linear conditional-mean model. It assumes the model is an adequate summary for these profiles and that the treatment and profile coding are as stated; no normality assumption is needed to calculate the fitted mean contrasts.
Worked Example: Prognostic Difference With a Small Treatment Interaction
Suppose \(H=0\) denotes a lower-risk profile and \(H=1\) a higher-risk profile. A fitted mean model for a hypothetical outcome measured in points is
where \(A=0\) denotes treatment A and \(A=1\) denotes treatment B. For the lower-risk profile (\(H=0\)), the fitted mean under A is \(80-15(0)+5(0)+2(0)(0)=80\) points. Under B it is \(80-15(0)+5(1)+2(1)(0)=85\) points. The fitted treatment contrast is \(85-80=5\) points.
For the higher-risk profile (\(H=1\)), the fitted mean under A is \(80-15(1)+5(0)+2(0)(1)=65\) points. Under B it is \(80-15(1)+5(1)+2(1)(1)=72\) points. Its fitted contrast is \(72-65=7\) points.
The higher-risk profile has lower fitted outcomes under both treatments: 65 rather than 80 points under A, and 72 rather than 85 points under B. That is a prognostic difference. The fitted treatment contrasts are 5 and 7 points, respectively, so the model’s interaction term corresponds to a 2-point difference in treatment contrasts. Whether this estimated difference is persuasive requires uncertainty assessment; the fitted values alone do not establish that the true effects differ.
This calculation also clarifies how model terms should be read. The \(-15H\) term shifts the expected outcome for the higher-risk profile under both treatments. It is prognostic. The \(2AH\) term changes the B-versus-A contrast when \(H=1\); it represents treatment-effect heterogeneity on the model’s additive mean scale. On another outcome scale, such as a ratio scale, the meaning of “same treatment effect” can differ, so the scale must be made explicit.
Translate Probabilities Into Expected Assignments
An allocation probability can describe a single next assignment or, under a stated set of fixed probabilities, the expected number assigned to an arm in a group. It does not promise an exact number. For an adaptive design, probabilities may change with history, so multiplying a group size by one probability is justified only when that probability applies to every assignment in the group, or when the calculation is explicitly an approximation using a common probability.
Worked Example: Expected Assignments Under Fixed Profile Probabilities
Consider a hypothetical planning calculation with 40 lower-risk and 20 higher-risk patients. For illustration, suppose the probability of B remains fixed at 0.55 for each lower-risk patient and 0.70 for each higher-risk patient. The expected number assigned to B among the lower-risk patients is
For the higher-risk patients it is
The expected total assigned to B is \(22+14=36\) of 60 patients. The corresponding overall expected proportion is \(36/60=0.60\), or 60%. The profile-specific expected proportions remain 55% and 70%; the overall value is their patient-count-weighted average, not their unweighted average.
These are expectations, not guaranteed counts. Even with the stated fixed probabilities, the realized number assigned to B can differ from 36. In an adaptive trial, the probabilities may also be updated as outcomes accrue, so the fixed-probability calculation is only appropriate for the explicitly stated illustration.
A Practical Reading Sequence
When reviewing estimated allocation probabilities by patient profile, use a consistent sequence. It helps prevent a design quantity from being interpreted as a treatment-effect estimate.
Confirm which treatment’s assignment probability is reported, and verify that probabilities across treatments sum to one.
Record the patient profile and trial history at which the probability was calculated. Comparisons across different histories combine profile and time/history differences.
Determine which estimated outcomes, exploration components, and design constraints contribute to the probability. Do not infer the rule from the probability table alone.
For each profile, calculate the predicted outcome difference between treatments on the stated outcome scale. Contrast differences, not outcome levels, address heterogeneity.
Probabilities close to zero or one mean assignments to one arm are uncommon for that profile under those conditions. Such probabilities can limit the available within-profile comparison.
Comparisons, Diagnostics, and Limitations
| Quantity | What it answers | What it does not establish by itself |
|---|---|---|
| Profile-specific allocation probability | How likely the design is to assign a treatment under specified conditions | That the treatment is more effective for that profile |
| Profile-specific predicted outcome | What outcome level the model predicts for a treatment and profile | How much the treatment changes the outcome relative to another treatment |
| Profile-specific treatment contrast | How the model’s predicted outcomes differ between treatments within a profile | That contrasts truly differ across profiles without uncertainty assessment |
| Observed assignment proportion | What fraction of observed patients in a group received an arm | The conditional probability for a particular next patient and history |
A useful diagnostic is to recompute or display probabilities at a common trial history for a set of clinically meaningful profiles, alongside the model-predicted outcomes under each treatment and their within-profile contrasts. Also verify that the profile labels, treatment coding, and outcome scale are consistent. If probabilities are presented without the rule or history that produced them, their interpretation is incomplete.
This comparison has limits. Model-based treatment contrasts are estimated quantities and can be uncertain, especially when few patients with a profile have received one of the treatments. Allocation probabilities can become very small for an arm, reducing the amount of direct information about that arm in that profile. A probability table does not replace uncertainty intervals for treatment contrasts, nor does it prove that the outcome model describes the relevant patients well. The allocation mechanism and the outcome model answer related but distinct questions.
Common Mistakes
- Wrong: “The higher-risk profile has a higher probability of B, so B must work better for those patients.” Correction: The probability describes assignment under the rule. Compare estimated B-versus-A outcome contrasts by profile to assess modeled effect differences.
- Wrong: “The higher-risk profile has worse predicted outcomes, so it has a smaller treatment effect.” Correction: Worse outcomes under both treatments can reflect prognosis alone. Subtract the treatment-specific predictions within each profile.
- Wrong: “A probability of 0.70 means exactly 70% of that profile will receive B.” Correction: It is a conditional assignment chance. Realized counts vary, and adaptive probabilities may change over time.
- Wrong: “Probabilities can be compared across profiles regardless of when they were calculated.” Correction: If the allocation rule uses trial history, hold the history fixed or acknowledge that history changed.
Check Your Understanding
Answer each question using the distinction between assignment probability, prognosis, and treatment contrast.
- For a profile, the estimated probability of B is 0.63. What is the estimated probability of A in a two-treatment design, and what does 0.63 describe?
- A model predicts outcomes of 40 and 46 points under A and B for one profile, and 30 and 36 points under A and B for another. Calculate both treatment contrasts. Is there fitted heterogeneity on the additive mean scale?
- For 30 patients, the probability of B is fixed at 0.40. What is the expected number assigned to B? Does the design guarantee that count?
- Why is it unsafe to conclude that treatment B benefits a profile more just because that profile has a higher estimated allocation probability for B?
Solutions. (1) The probability of A is \(1-0.63=0.37\). The 0.63 is the conditional chance of assigning B for that profile and the specified trial history; it is not a response probability. (2) The first contrast is \(46-40=6\) points; the second is \(36-30=6\) points. The fitted contrasts are equal, so there is no fitted heterogeneity on this additive mean scale, although uncertainty remains relevant. (3) The expected number is \(30(0.40)=12\). This is an expectation, not a guaranteed realized count. (4) Assignment probabilities are determined by the allocation rule and its inputs, which may include factors besides treatment contrasts. A higher probability of B does not establish greater benefit; compare the within-profile predicted contrasts and assess their uncertainty.