Power Under The Null And Alternative

Two Rejection Probabilities, Two Different Questions

A trial design is not judged by a single rejection rate. The probability that a test rejects when the null hypothesis is true describes false-positive behavior. The probability that it rejects when a specified treatment effect is present describes power. These probabilities answer different questions, so a design comparison should report and interpret them separately.

This distinction matters when comparing fixed randomization with response-adaptive randomization. A design that rejects more often under a treatment effect may have greater power, but only if its rejection behavior under the null is also acceptable. Otherwise, an apparent power gain could partly reflect a test that rejects too readily, including when there is no treatment effect. The earlier material on Type I error as a fairness check motivates examining this null behavior directly.

Definition: For a specified test and a specified data-generating model, the rejection probability is the probability that the test rejects \(H_0\). Under a particular null model, this is the test's rejection probability under the null; for a level-\(\alpha\) test, it should be no greater than \(\alpha\) under the null models for which level control is claimed. Under a specified alternative model, the rejection probability is the power at that alternative. The probability of not rejecting under that alternative is the type II error probability, often denoted \(\beta\), so power is \(1-\beta\).

Power is not one number that applies to every possible treatment effect. It depends on the alternative being considered, as well as on the sample size, allocation, outcome variability, test, significance level, and other assumptions. A power statement therefore needs a specified alternative, such as a mean difference of \(0.4\) outcome units. It is not the probability that the alternative is true, nor does a high power value establish that the alternative is true.

A Normal-Mean Test as a Reference Case

For a transparent reference calculation, suppose independent outcomes in two groups are normally distributed with a common, known standard deviation \(\sigma\). Let \(n_T\) and \(n_C\) be the fixed sample sizes, and let the treatment effect be \(\delta=\mu_T-\mu_C\). The estimator \(\bar{x}_T-\bar{x}_C\) has standard error \(\sigma\sqrt{1/n_T+1/n_C}\). Consider the two-sided, known-variance \(z\)-test of \(H_0:\delta=0\) against \(H_a:\delta\ne0\), at significance level \(\alpha\). These assumptions justify the standard normal test statistic; they should not be presumed for an adaptive design without checking its analysis and calibration.

SE=\sigma\sqrt{\frac{1}{n_T}+\frac{1}{n_C}}, \qquad Z=\frac{\bar{x}_T-\bar{x}_C}{SE}.

The test rejects when \(|Z|>z_{1-\alpha/2}\), where \(z_{1-\alpha/2}\) is the \(1-\alpha/2\) quantile of the standard normal distribution. Under the null model \(\delta=0\), \(Z\sim N(0,1)\), and the two rejection tails together have probability \(\alpha\). Under a specified alternative \(\delta\), \(Z\sim N(\lambda,1)\), where \(\lambda=\delta/SE\). Thus the probability of exceeding the upper critical value and the probability of falling below the lower critical value give the power:

P(|Z|>z_{1-\alpha/2}\mid \delta) = 1-\Phi\!\left(z_{1-\alpha/2}-\lambda\right) +\Phi\!\left(-z_{1-\alpha/2}-\lambda\right).

Here \(\Phi\) is the standard normal cumulative distribution function. The notation conditions the calculation on the specified model and effect; it does not mean that the effect itself is uncertain. For a two-sided test, power depends on the magnitude of \(\delta\), and the formula includes both rejection tails. When the effect points strongly in one direction, the opposite tail is often very small, but it is still part of the two-sided probability.

Worked Calculations

Worked Example: Null Rejection Probability and Power With Equal Allocation

Suppose, hypothetically, that a two-arm fixed-randomization trial has \(n_T=n_C=100\), normally distributed outcomes, and known \(\sigma=1\) outcome unit in each arm. Use the two-sided \(z\)-test above at \(\alpha=0.05\). Under the null, the model gives \(Z\sim N(0,1)\), so

P(\text{reject }H_0\mid \delta=0) =P(|Z|>1.960)=0.0500.

Now specify the alternative \(\delta=0.4\) outcome units. The standard error and noncentrality parameter are

SE=1\sqrt{\frac{1}{100}+\frac{1}{100}} =\sqrt{0.0200}=0.1414, \qquad \lambda=\frac{0.4}{\sqrt{0.0200}}\approx 2.8284.

Therefore,

\begin{aligned} \text{Power} &=1-\Phi\!\left(1.960-\frac{0.4}{\sqrt{0.0200}}\right) +\Phi\!\left(-1.960-\frac{0.4}{\sqrt{0.0200}}\right)\\ &\approx 0.8074. \end{aligned}

In this hypothetical model, the test rejects in about 80.74% of repeated trials when the true mean difference is \(0.4\) outcome units. Its rejection probability under the null is 5.00%. Those figures describe distinct operating characteristics of this test under two different models.

Worked Example: Comparing Power After Changing the Allocation

Keep the outcome model, total sample size of 200, significance level, test, and specified effect from the previous example, but hypothetically use \(n_T=75\) and \(n_C=125\). The standard error becomes

SE=1\sqrt{\frac{1}{75}+\frac{1}{125}} =\sqrt{0.0213333}=0.1461.

At \(\delta=0.4\) outcome units, the noncentrality parameter is \(0.4/\sqrt{\frac{1}{75}+\frac{1}{125}}=2.7386\). Substitution into the same two-sided power formula gives

\begin{aligned} \text{Power} &=1-\Phi(1.960-2.7386)+\Phi(-1.960-2.7386)\\ &=\Phi(0.7786)+\Phi(-4.6986)\\ &\approx 0.7819. \end{aligned}

The calculated power is about 78.19%, compared with 80.74% for equal allocation, a difference of about 2.55 percentage points. Under the stated known-variance test, the null rejection probability remains 0.0500: the null distribution of \(Z\) is standard normal for either allocation. This example separates a change in power at the chosen alternative from a change in false-positive behavior. It does not establish that one allocation is preferable for every outcome model or study objective.

Justification: Under the stated independent normal model, the difference in sample means has mean \(\delta\) and variance \(\sigma^2(1/n_T+1/n_C)\). Dividing by its standard deviation gives a normal statistic with mean \(\lambda=\delta/SE\) and variance 1. Rejection occurs in the two disjoint events \(Z>z_{1-\alpha/2}\) or \(Z<-z_{1-\alpha/2}\). Their probabilities are respectively \(1-\Phi(z_{1-\alpha/2}-\lambda)\) and \(\Phi(-z_{1-\alpha/2}-\lambda)\); adding them yields the power expression.

Worked Example: Reading Simulated Rejection Rates

Suppose a hypothetical simulation compares fixed and response-adaptive designs using the same nominal \(\alpha=0.05\), the same prespecified test, and 10,000 independent simulated trials per scenario. Under the null model, the test rejects in 530 fixed-design trials and 490 adaptive-design trials. Under a specified alternative model, it rejects in 7,980 fixed-design trials and 7,610 adaptive-design trials. The estimated rejection proportions are \(530/10{,}000=0.053\) and \(490/10{,}000=0.049\) under the null, and \(7{,}980/10{,}000=0.798\) and \(7{,}610/10{,}000=0.761\) under the alternative.

The first pair estimates null rejection probabilities, not power. The second pair estimates power at the particular specified alternative. To gauge simulation noise, the Monte Carlo standard error of an estimated rejection proportion \(\hat q\) from \(R\) independent simulated trials is approximately \(\sqrt{\hat q(1-\hat q)/R}\). For the two power estimates, these are

\sqrt{\frac{0.798(0.202)}{10{,}000}}=0.00401, \qquad \sqrt{\frac{0.761(0.239)}{10{,}000}}=0.00426.

The estimated power difference is \(0.798-0.761=0.037\), or 3.7 percentage points. If the simulations are independent, the Monte Carlo standard error of that difference is approximately \(\sqrt{0.00401^2+0.00426^2}=0.00586\). This quantifies simulation noise under the chosen models, not uncertainty about whether those models represent a future trial. The null estimates also deserve scrutiny: their difference is \(0.004\), and the approximate standard error of that difference is \(\sqrt{0.053(0.947)/10{,}000+0.049(0.951)/10{,}000}=0.00311\). The estimates alone do not show a clear difference in null rejection rates, but neither should a single simulation be treated as exact.

How to Compare Designs Fairly

A meaningful design comparison evaluates rejection probabilities under matched conditions. In particular, specify the test, the significance level, the null model, and the alternative model before interpreting the rates. The alternative should state the effect size and any other outcome features needed to define the data-generating model. If the designs use different tests or estimators, describe that difference rather than attributing every rejection-rate difference to randomization alone.

Scenario	Quantity to estimate or calculate	Interpretation
Null model	Probability of rejecting \(H_0\)	False-positive behavior; assess against the claimed significance level
Specified alternative	Probability of rejecting \(H_0\)	Power at that particular alternative
Specified alternative	Probability of not rejecting \(H_0\)	Type II error probability at that alternative, equal to \(1-\text{power}\)

For a fixed design, a justified analytic distribution may make these probabilities calculable directly. For a response-adaptive design, the allocation depends on accumulating outcomes, so a reference formula for fixed allocation need not give the correct distribution of the final statistic. Simulation can estimate null rejection and power if each simulated trial reproduces the allocation rule, outcome model, stopping rules, missing-data handling, and analysis procedure intended for the comparison. The test must be applied to simulated data in the same way it would be applied in the trial.

Check null calibration across the null models relevant to the claim, not only at one convenient parameter setting when the null is composite. A level-\(\alpha\) procedure controls the rejection probability at or below \(\alpha\) for the null cases covered by its guarantee. An estimate somewhat above \(\alpha\) in a finite simulation may reflect Monte Carlo variation; a larger or persistent excess may indicate poor calibration, a coding error, or a mismatch between the test assumptions and the simulated design. Increasing the number of simulations can reduce Monte Carlo error, but it cannot repair a misspecified outcome model or an invalid test.

Fix the comparison.
State the designs, endpoint, test, significance level, and analysis rules being compared.

Evaluate null behavior.
Generate data under the null model and estimate or derive the rejection probability. Compare it with the claimed level and account for Monte Carlo error if simulated.

Evaluate specified alternatives.
Repeat under clearly specified treatment effects. Report power separately for each alternative and describe the outcome units.

Common Mistakes and Limitations

Calling the null rejection rate “power.” Power is calculated under an alternative. A rejection under the null is a false positive, and its probability concerns type I error.
Reporting power without naming the effect. “Power is 80%” is incomplete unless the alternative and its assumptions are stated. Power can differ substantially across treatment effects.
Calling a higher rejection rate a design improvement without checking the null. A test can reject more often under both null and alternative because its calibration is poor. Compare null behavior as well as power.
Treating simulation estimates as exact probabilities. A finite number of simulated trials produces Monte Carlo error. Report the number of replicates and, where useful, an estimate of that error.
Using a fixed-allocation formula automatically for adaptive allocation. The formula above relies on its stated normal model and fixed sample sizes. Use it for an adaptive design only if the distributional justification applies; otherwise evaluate the actual design and analysis, often by simulation.

These rejection probabilities describe repeated-trial properties under specified assumptions. They do not guarantee the result of a particular trial, quantify patient benefit directly, or establish that one design is universally better. Analytic calculations can be precise under a correct model; simulation can represent more complicated rules, but its conclusions remain limited by the scenarios and assumptions simulated.

Check Your Understanding

Answer each question by identifying the model under which the rejection probability is being evaluated.

A test rejects in 5% of simulated trials under \(H_0\) and 72% under a specified alternative. Which number estimates power?
At a specified alternative, a test has power 0.80. What is the type II error probability at that alternative?
In the equal-allocation example, what were the null rejection probability and power at a mean difference of \(0.4\) outcome units?
Why is a higher simulated rejection proportion under the alternative not, by itself, enough to conclude that a design is better?
With 10,000 simulations and estimated power 0.798, use the approximate Monte Carlo standard error formula to find the simulation error scale.

Solutions. 1. The 72% rejection rate under the specified alternative estimates power; 5% is the null rejection rate. 2. The type II error probability is \(1-0.80=0.20\). 3. The null rejection probability was 0.0500 and the power was approximately 0.8074. 4. The null rejection rate must also be examined to check that the test is not gaining rejections through inflated false-positive behavior; the comparison also depends on matched assumptions and analysis. 5. The standard error is \(\sqrt{0.798(0.202)/10{,}000}\approx0.00401\), about 0.40 percentage points.