Benefit-Harm Thresholds With Unequal Clinical Costs

Why Unequal Clinical Costs Change the Design

A two-sided sequential trial may stop early to declare benefit or to declare harm. Those conclusions need not have equal consequences. A false benefit claim might lead to widespread adoption of an ineffective treatment, while a false harm claim might prevent use of a genuinely beneficial treatment. An asymmetric design can reflect this difference by setting distinct error budgets and distinct stopping limits for the two directions.

The clinical costs do not, by themselves, dictate a unique statistical design. A design also needs an explicit policy for translating those costs into acceptable false-decision risks. The construction here makes that policy visible: choose directional error budgets, state how they relate to the cost priorities, and calibrate the sequential boundaries to those budgets. This separates a clinical judgment from the mathematics that controls error.

Definition: Let \(\theta\) denote the treatment effect, with positive values representing benefit and negative values representing harm. Let \(\delta_B\geq 0\) be the minimum effect regarded as clinically important benefit, and let \(\delta_H\geq 0\) be the minimum magnitude regarded as clinically important harm. A false benefit conclusion occurs when benefit is declared although \(\theta\leq\delta_B\); a false harm conclusion occurs when harm is declared although \(\theta\geq-\delta_H\). The directional error budgets \(\alpha_B\) and \(\alpha_H\) bound the probabilities of these respective false conclusions.

The two clinical thresholds need not be equal. They describe what effects matter clinically; they are not significance levels and are not chosen merely to balance the statistical tails. At interim look \(k\), suppose the estimate is \(\hat{\theta}_k\) with known standard error \(SE_k\). The evidence measured from the benefit threshold and harm threshold is \(Z_{B,k}=(\hat{\theta}_k-\delta_B)/SE_k\) and \(Z_{H,k}=(\hat{\theta}_k+\delta_H)/SE_k\). A benefit claim requires unusually large \(Z_{B,k}\); a harm claim requires unusually small \(Z_{H,k}\).

From Unequal Costs to Error Budgets

Suppose the design team assigns relative costs \(C_B>0\) to a false benefit conclusion and \(C_H>0\) to a false harm conclusion. One transparent policy is to choose the budgets so the two cost-weighted error exposures are equal: \(C_B\alpha_B=C_H\alpha_H\), subject to \(\alpha_B+\alpha_H=\alpha\), where \(\alpha\) is the total budget available for the two directional errors. Solving these equations gives:

\alpha_B=\alpha\frac{C_H}{C_B+C_H}, \qquad \alpha_H=\alpha\frac{C_B}{C_B+C_H}.

Under this policy, the direction with greater false-decision cost receives the smaller error budget. This is a chosen allocation rule, not a universal optimality theorem. A decision maker could instead impose a maximum risk for one direction, use a utility model, or choose budgets based on other constraints. Whatever the policy, it should be stated before the boundaries are constructed.

Worked Example: Allocating Error by Relative Clinical Cost

Suppose the total error budget is \(\alpha=0.020\). For illustration, a false benefit claim is assigned cost \(C_B=4\) units and a false harm claim cost \(C_H=1\) unit. Use the equal-cost-weighted-exposure policy.

The benefit budget is \(\alpha_B=0.020(1)/(4+1)=0.004\). The harm budget is \(\alpha_H=0.020(4)/(4+1)=0.016\). Their sum is \(0.004+0.016=0.020\). The weighted exposures are also equal: \(4(0.004)=0.016\) and \(1(0.016)=0.016\).

Thus this policy tolerates a smaller probability of a false benefit claim than of a false harm claim, reflecting the larger assigned cost of the former. The numbers are hypothetical policy inputs, not empirical estimates of clinical harm.

Constructing the Sequential Stopping Rule

Assume a prespecified set of \(K\) interim looks, a Gaussian \(Z\)-statistic with known standard errors at each look, and an estimator whose distribution is centered at \(\theta\). For the construction below, each marginal statistic has variance one after standardization. The looks may be dependent, as is usual when observations accumulate; the simple calibration uses a union bound and does not require independence. The look schedule and analysis plan must be fixed in advance.

For a simple, conservative calibration, divide each directional budget across the looks. Choose nonnegative values \(\alpha_{B,k}\) and \(\alpha_{H,k}\) such that \(\sum_{k=1}^K\alpha_{B,k}\leq\alpha_B\) and \(\sum_{k=1}^K\alpha_{H,k}\leq\alpha_H\). Let \(z_q\) be the \(q\)-quantile of the standard normal distribution. At look \(k\), define the upper and lower evidence limits as \(c_{B,k}=z_{1-\alpha_{B,k}}\) and \(c_{H,k}=z_{1-\alpha_{H,k}}\). Stop for benefit if \(Z_{B,k}\geq c_{B,k}\); stop for harm if \(Z_{H,k}\leq-c_{H,k}\). If neither condition holds, continue, unless the final look has been reached.

\begin{aligned} \text{Benefit stop at look }k &\quad\Longleftrightarrow\quad \hat{\theta}_k\geq\delta_B+c_{B,k}SE_k,\\ \text{Harm stop at look }k &\quad\Longleftrightarrow\quad \hat{\theta}_k\leq-\delta_H-c_{H,k}SE_k. \end{aligned}

These equivalent effect-scale expressions show the role of each ingredient. The clinically important thresholds shift the targets away from zero, while the statistical limits determine how much evidence beyond those targets is required. Because the two budgets can differ, the evidence limits can differ as well.

Justification: At a benefit-null boundary \(\theta=\delta_B\), \(Z_{B,k}\) has a standard normal distribution. For any \(\theta\leq\delta_B\), its mean is no greater than zero, so \(P(Z_{B,k}\geq c_{B,k})\leq\alpha_{B,k}\). The probability of at least one false benefit crossing is therefore at most the sum of the per-look probabilities, which is no greater than \(\alpha_B\). The same argument applied to \(-Z_{H,k}\) gives a false harm probability no greater than \(\alpha_H\) for \(\theta\geq-\delta_H\). This union-bound argument remains valid when the looks are dependent, although it may be conservative.

Worked Example: Two-Look Limits Under Unequal Budgets

Continue the hypothetical design above, with two looks and equal division within each directional budget. Then \(\alpha_{B,k}=0.004/2=0.002\), and \(\alpha_{H,k}=0.016/2=0.008\) at each look. The corresponding standard normal limits are \(z_{0.998}\approx2.878\) and \(z_{0.992}\approx2.409\). Thus the benefit limit is \(2.878\) and the magnitude of the harm limit is \(2.409\) at each look.

Suppose \(\delta_B=2\) outcome units, \(\delta_H=1.5\) outcome units, and at the first look \(\hat{\theta}_1=4.9\) units with \(SE_1=1.0\) unit. Then \(Z_{B,1}=(4.9-2)/1.0=2.900\), which exceeds \(2.878\). The effect-scale benefit threshold is \(2+2.878(1.0)=4.878\) units, and the estimate \(4.9\) units exceeds it. The rule stops for benefit at look 1. It does not declare that the probability the treatment is beneficial is 99.8%; it makes a sequential decision under the specified error-control plan.

As a separate hypothetical observation illustrating the lower direction, if \(\hat{\theta}_1=-4.0\) units with the same standard error, then \(Z_{H,1}=(-4.0+1.5)/1.0=-2.500\). Since \(-2.500\leq-2.409\), the harm boundary is crossed. The effect-scale harm limit is \(-1.5-2.409(1.0)=-3.909\) units, and \(-4.0\) is below it.

Worked Example: Continuing at the First Look and Stopping at the Second

Use the same thresholds, with \(\delta_B=2\) units and equal per-look limits \(2.878\) for benefit and \(2.409\) for harm. At look 1, suppose \(\hat{\theta}_1=4.7\) units and \(SE_1=1.0\) unit. Then \(Z_{B,1}=(4.7-2)/1.0=2.700\), below \(2.878\); and \(Z_{H,1}=(4.7+1.5)/1.0=6.200\), not at or below \(-2.409\). The trial continues.

At look 2, suppose the updated estimate is \(4.6\) units and \(SE_2=0.7\) unit. Then \(Z_{B,2}=(4.6-2)/0.7=3.714\), which exceeds \(2.878\). Equivalently, the benefit effect-scale limit is \(2+2.878(0.7)=4.0146\) units, and \(4.6\) exceeds it. The rule stops for benefit at look 2. The different estimate at the second look is possible because it uses additional accumulating data; it does not invalidate the prespecified rule.

What the Rule Guarantees—and What It Does Not

The error statements are directional and conditional on their corresponding null regions: under any \(\theta\leq\delta_B\), the chance of ever crossing a benefit boundary is at most \(\alpha_B\); under any \(\theta\geq-\delta_H\), the chance of ever crossing a harm boundary is at most \(\alpha_H\). If the two null regions overlap, as they generally do when the thresholds define a neutral range, both guarantees apply in that range. With \(\alpha_B+\alpha_H\leq\alpha\), the probability of either false directional claim in the overlap is at most \(\alpha\), by adding the two bounds.

The simple allocation across looks is not usually the most efficient calibration. It ignores the joint distribution of the sequential statistics and can produce more stringent limits than necessary. A group sequential calibration that uses their dependence can often spend the same directional budgets with less conservatism. The cost policy still determines the budgets; a different calibration method determines the boundaries that achieve them. The Gaussian known-standard-error assumptions must also be credible, or an appropriate model-specific calibration must replace the normal calculation.

A useful comparison is with three alternatives. A symmetric two-sided rule imposes equal directional limits, which may not match unequal clinical priorities. Unequal tail error allocation can reflect directional priorities but does not itself encode different clinical effect thresholds. Effect-scale thresholds define what counts as meaningful benefit or harm, but do not alone control the chance of false sequential claims. The construction here makes both choices explicit: separate clinical thresholds and directional error budgets.

Diagnostics and Common Pitfalls

Cost is not a probability. A cost ratio cannot be substituted directly for a tail probability. State the allocation policy that translates cost priorities into budgets.
Do not confuse clinical thresholds with boundary limits. \(\delta_B\) and \(\delta_H\) are effect-scale quantities; \(c_{B,k}\) and \(c_{H,k}\) are standardized evidence limits.
Do not treat looks as independent. The union-bound construction does not need independence. Other calibrations must use the correct joint sequential distribution rather than multiply single-look probabilities as if looks were independent.
Check the direction and units. Benefit uses \(\hat{\theta}_k-\delta_B\); harm uses \(\hat{\theta}_k+\delta_H\). The effect-scale limits have the same units as \(\theta\), while \(Z\)-limits are unitless.
Do not interpret no crossing as proof of no effect. It means only that the prespecified stopping evidence was not reached. The trial continues or ends without that directional conclusion, according to its plan.

Key takeaway: Unequal clinical consequences can motivate unequal directional error budgets, but only through a stated decision policy. Apply those budgets to sequential evidence measured from the relevant clinical effect thresholds, and verify that the resulting rule controls each directional error under its stated assumptions.

Check Your Understanding

Answer each question using the definitions and construction above. Brief solutions follow.

If \(\alpha=0.030\), \(C_B=2\), and \(C_H=1\), what budgets result from equal cost-weighted error exposure?
With two equally divided looks and a benefit budget of \(0.006\), what is the per-look benefit tail probability?
For \(\delta_H=1\) unit, \(SE_k=0.5\) unit, and \(c_{H,k}=2.0\), state the harm stopping threshold on the effect scale.
Why can the union-bound justification apply when the interim statistics are dependent?
What does failure to cross either boundary at an interim look imply?

Solutions. (1) \(\alpha_B=0.030(1)/(2+1)=0.010\) and \(\alpha_H=0.030(2)/(2+1)=0.020\); the weighted exposures are both \(0.020\). (2) \(0.006/2=0.003\) at each look. (3) The threshold is \(-\delta_H-c_{H,k}SE_k=-1-2(0.5)=-2\) units; stop for harm when the estimate is at or below \(-2\) units. (4) The probability of a union of crossing events is at most the sum of their marginal probabilities, regardless of dependence. (5) The rule continues to the next scheduled look, if one remains; it does not establish that there is no benefit or harm.