At-Least-K Power Criteria

From Pairwise Results to a Count of Successful Comparisons

A multi-arm trial can be designed to answer a question about the number of treatments that succeed, rather than requiring success from every comparison or counting any single success as sufficient. For example, a decision rule might require at least two of three treatment-control comparisons to meet a prespecified criterion. The relevant operating characteristic is then the probability that the number of successful comparisons reaches that threshold.

The success rule must be specified before calculating its power. It includes which comparisons are counted and what qualifies as success for each comparison, such as rejection under a chosen testing procedure. Power is evaluated under a specified configuration of treatment effects; it is not the probability that a treatment is effective or that the null hypothesis is true.

This definition depends on the joint behaviour of the success events. If the events are independent, their probabilities can be combined using an independence calculation. If they are dependent, the joint distribution must be accounted for. In a shared-control trial, for instance, comparisons may have related test statistics because they use the same control group. Dependence does not change the definition; it changes how its probability is calculated.

The success events should also match the planned analysis. If success means a comparison rejects using a multiplicity-adjusted procedure, \(S_i\) must refer to rejection under that procedure, not under an unadjusted test substituted for convenience. The event probabilities used below are hypothetical values assumed to come from the specified procedure under the stated effect configuration.

Calculation When Comparisons Are Independent

Suppose the \(m\) success indicators are independent under the treatment-effect configuration of interest, and write \(p_i=P(S_i)\) for the power of comparison \(i\). The number of successes \(N\) is then a sum of independent Bernoulli random variables, which need not have the same success probability. Its probability-generating function is

To see why, \(z^N=z^{I_1+\cdots+I_m}=\prod_i z^{I_i}\). Independence gives \(E(z^N)=\prod_i E(z^{I_i})\), and \(E(z^{I_i})=(1-p_i)+p_i z\) because \(I_i\) is either 0 or 1. The coefficient of \(z^r\) in \(G_N(z)\) is \(P(N=r)\): each factor contributes either a failure term or a success term, and the product’s \(z^r\) terms correspond to exactly \(r\) successes. Therefore,

where \([z^r]G_N(z)\) means the coefficient of \(z^r\). This gives a direct calculation for small numbers of comparisons and a convenient computational method for larger \(m\). When every comparison has the same power \(p\), the number of successes has a binomial distribution and \(P(N\geq k)=\sum_{r=k}^{m}\binom{m}{r}p^r(1-p)^{m-r}\). Equal marginal powers alone do not imply this binomial model; independence is also required.

Calculation When Success Events Are Dependent

Without independence, the product of marginal powers does not generally give the probability of a pattern of successes. A general expression sums the joint probabilities of every success pattern containing at least \(k\) successes. For a subset \(A\) of \(\{1,\ldots,m\}\), let the event that exactly the comparisons in \(A\) succeed be

The patterns are mutually exclusive and exhaustive. Therefore, at-least-\(k\) power is the sum of the probabilities of those patterns for which \(|A|\geq k\). This is exact if the required joint probabilities are known, though enumeration involves \(2^m\) patterns. For a small number of comparisons, listing the patterns is transparent; for larger problems, a joint model or simulation may be more practical.

Checking Assumptions and Choosing a Calculation

The calculation method is only as credible as the probabilities and assumptions it uses. Before reporting at-least-\(k\) power, check the following:

• Counted comparisons: verify that \(m\) and the comparisons included in \(N\) match the protocol’s success rule.
• Success definition: specify the decision rule for each comparison, including any multiplicity adjustment. Do not substitute a different test’s rejection probability.
• Effect configuration: state the treatment effects under which power is evaluated. Pairwise powers can change across configurations.
• Dependence: use independence only when justified by the design and analysis model. If comparisons share data or otherwise have related outcomes, use their joint distribution or an appropriate joint model.
• Probability calibration: ensure the marginal or joint probabilities are computed under the same analysis rule, allocation, and assumptions as the planned trial.

The probability-generating-function product is an efficient alternative to listing every pattern, but it rests on independence. Exact pattern enumeration is a direct alternative when the joint distribution is available and \(m\) is small. For dependent tests whose joint distribution is not available in closed form, one can simulate trial outcomes under the specified model, apply the full analysis rule to each simulated trial, count the proportion with \(N\geq k\), and report the Monte Carlo uncertainty as well. Simulation does not remove the need to specify a credible joint model.

A basic diagnostic is to check that the calculated probability lies between 0 and 1 and that all probabilities in a stated joint distribution sum to 1. For an independent calculation, check that the \(p_i\) values lie between 0 and 1 and that the polynomial’s coefficients correspond to the possible success counts. A further check is monotonicity: for a fixed joint model, lowering the threshold \(k\) cannot decrease \(P(N\geq k)\). If the calculation violates these checks, revisit the event definitions, arithmetic, or implementation.

Common Pitfalls and Practical Limits

A frequent error is to report an individual comparison’s power as though it were the probability of meeting a multi-comparison count rule. Another is to calculate the independent formula from marginal powers even though the success events are dependent. In the joint-distribution example, the at-least-two probability was 0.50. Multiplying marginal probabilities as if independent would answer a different model’s question, not the one specified by that joint distribution.

There is also a distinction between the event counted and the scientific claim. An at-least-\(k\) testing success rule counts comparisons meeting the statistical criterion. It does not, by itself, establish that exactly \(k\) treatments have clinically meaningful effects, nor does its power state the probability that a treatment is truly effective. Such interpretations require the relevant scientific definitions and the assumed effect configuration.

The approach is flexible, but it does not select \(k\), define a clinically meaningful effect, or determine a valid multiplicity procedure. Those are design choices that must precede the power calculation. Its main value is to make the operating target explicit: the probability that the chosen analysis produces at least the required number of successes.