Estimating Power by Simulation

Why Estimate Power by Simulation?

Power is the probability that a specified test rejects \(H_0\) when a particular alternative model is true. It depends on more than the sample size: the assumed effect, variability, significance level, test direction, and test procedure all matter. For some simple tests, formulas give power directly. But for a more complicated design or rejection rule, a convenient formula may be unavailable or difficult to evaluate. Simulation offers a direct approach: repeatedly generate data as if the specified alternative were true, apply the planned test, and count how often it rejects.

A simulation estimates power only for the model and procedure that are actually simulated. It does not discover the true power for every possible alternative, and it does not repair a poorly chosen statistical model. The assumptions and the rejection rule must therefore be set before generating the simulated data.

Definition: For a specified alternative data-generating model, the power of a test is the probability that the test rejects \(H_0\). To estimate this power by simulation, generate \(R\) independent data sets from that alternative model, apply the same prespecified test to each one, and divide the number of rejections by \(R\). The resulting fraction is the simulation estimate of power.

The Simulation Procedure

A useful simulation imitates the planned study and its analysis as faithfully as possible. Before starting, specify the sample size, the data-generating model under the alternative of interest, the significance level, and the exact decision rule. The decision rule includes the statistic, the rejection threshold, and whether the test is one-sided or two-sided. Those choices stay fixed across all replications.

For each replication, generate a complete hypothetical data set under the alternative. Calculate the test statistic and mark the replication as a rejection if, and only if, the prespecified rule says to reject \(H_0\). After \(R\) independent replications, let \(I_r=1\) if replication \(r\) rejects and \(I_r=0\) otherwise. The estimated power is the average of these indicators:

\widehat{\text{Power}}=\frac{1}{R}\sum_{r=1}^{R} I_r =\frac{\text{number of simulated rejections}}{R}.

The estimate is a proportion, so it ranges from 0 to 1. For example, an estimate of 0.80 means that 80% of the simulated data sets led to rejection under the specified alternative. It does not mean that a single future study has an 80% probability of producing a particular result unless its design and assumptions match those used in the simulation.

Specify the alternative model.
Choose the parameter value or other data-generating conditions at which power is to be estimated. State the model assumptions, such as independent observations and a common population distribution.

Fix the planned test.
Set the hypotheses, significance level, test statistic, and rejection rule. Do not alter the rule in response to simulated results.

Generate and analyze replicates.
For each replication, generate a new data set under the alternative and record whether the fixed test rejects.

Summarize the rejections.
Divide the rejection count by the total number \(R\) of independent replications, and report the simulation effort and assumptions alongside the estimate.

Why the Rejection Fraction Estimates Power

Under the chosen alternative model, each simulated replication either rejects or does not reject. If the data sets are generated independently and the same test is applied each time, the indicators \(I_1,\ldots,I_R\) are independent Bernoulli random variables with success probability equal to the power, which we denote by \(\pi\). Thus \(E(I_r)=\pi\) and \(\operatorname{Var}(I_r)=\pi(1-\pi)\).

The average of the indicators has expected value \(\pi\), so the rejection fraction targets the desired power. Its variance also shows why estimates from a finite number of simulations fluctuate:

E\!\left(\widehat{\text{Power}}\right) =\frac{1}{R}\sum_{r=1}^{R}E(I_r) =\frac{1}{R}(R\pi) =\pi,

\operatorname{Var}\!\left(\widehat{\text{Power}}\right) =\operatorname{Var}\!\left(\frac{1}{R}\sum_{r=1}^{R}I_r\right) =\frac{1}{R^2}\sum_{r=1}^{R}\operatorname{Var}(I_r) =\frac{\pi(1-\pi)}{R}.

The simulation standard error is therefore estimated by substituting the observed rejection fraction for the unknown \(\pi\):

SE_{\text{simulation}} \approx \sqrt{\frac{\widehat{\text{Power}}\left(1-\widehat{\text{Power}}\right)}{R}}.

This standard error describes Monte Carlo variation: how much the estimate may vary because only \(R\) random replications were run. It is not the standard error of a treatment effect in one real study. Increasing \(R\) reduces this numerical uncertainty, at a rate proportional to \(1/\sqrt{R}\). For instance, multiplying the number of replications by four approximately halves the simulation standard error.

Worked Example: A Small Simulation for a Mean Test

Suppose a planned one-sided \(z\) test compares \(H_0:\mu=50\) with \(H_a:\mu>50\), using significance level \(\alpha=0.05\). Assume independent observations from a normal population with known \(\sigma=10\), and plan to sample \(n=25\). Estimate power at the specified alternative \(\mu=55\).

Check the model and test rule. For this example, the simulation assumes the observations are independent and normally distributed, with standard deviation 10 and mean 55 under the alternative. The null standard error of \(\bar{x}\) is \(\sigma/\sqrt{n}=10/\sqrt{25}=2\). The fixed upper-tail test rejects when \(Z>1.645\), equivalently when \(\bar{x}>50+1.645(2)=53.29\).

Apply the rule to simulated results. Imagine that 12 independent sample means generated under \(\mu=55\) are 54.1, 52.8, 55.6, 53.4, 56.2, 54.7, 51.9, 53.0, 57.1, 54.0, 52.6, and 55.2. The values above 53.29 are 54.1, 55.6, 53.4, 56.2, 54.7, 57.1, 54.0, and 55.2, giving 8 rejections.

Therefore the estimated power from this small illustrative run is \(8/12=0.667\), or about 66.7%. The estimate is rough because only 12 replications were used; a larger run would reduce Monte Carlo variation. Each sample mean must come from a newly generated sample under the specified alternative, not from repeatedly analyzing the same data.

Worked Examples With More Replications

Worked Example: Simulating Power for a Proportion Test

Consider a one-sided test of \(H_0:p=0.30\) versus \(H_a:p>0.30\), with \(\alpha=0.05\) and \(n=40\). Estimate power at the alternative \(p=0.45\) using the one-sample proportion \(z\) test. Assume each observation is an independent Bernoulli outcome with the same probability \(p\).

Fix the rejection rule first. Under the null, the standard error used by this test is \( \sqrt{0.30(1-0.30)/40}=\sqrt{0.00525}\approx0.07246 \). The test rejects when \( Z=(\hat{p}-0.30)/0.07246>1.645 \). This is equivalent to \(\hat{p}>0.30+1.645(0.07246)\approx0.4192\). Since \(\hat{p}=X/40\), where \(X\) is the number of successes, the rule is to reject when \(X\geq17\).

Simulate under the alternative. For each replication, generate 40 independent Bernoulli outcomes with success probability 0.45, calculate \(X\), and record a rejection if \(X\geq17\). Suppose a hypothetical run of \(R=2{,}000\) replications produces 1,476 rejections. The estimated power is

\widehat{\text{Power}}=\frac{1{,}476}{2{,}000}=0.738.

The simulation standard error is \( \sqrt{0.738(1-0.738)/2{,}000} =\sqrt{0.193356/2{,}000} =\sqrt{0.000096678} \approx0.00983 \). Thus the estimated power is 73.8%, with Monte Carlo standard error about 0.98 percentage points. This uncertainty concerns the finite simulation; it does not account for whether the assumed success probability or Bernoulli model is realistic.

Worked Example: Quantifying Simulation Precision

Suppose a two-sided \(z\) test is planned for \(H_0:\mu=100\) versus \(H_a:\mu\ne100\), with \(\alpha=0.05\), \(n=25\), and known \(\sigma=10\). Assume independent normal observations. To estimate power at \(\mu=104.64\), the test rejects when \(|Z|>1.96\).

The standard error of the sample mean is \(10/\sqrt{25}=2\), so the equivalent rule is to reject when \(\bar{x}<100-1.96(2)=96.08\) or \(\bar{x}>100+1.96(2)=103.92\). Generate independent samples of size 25 from a normal population with mean 104.64 and standard deviation 10, and apply these same cutoffs in every replication.

Imagine that \(R=10{,}000\) simulated samples produce 6,382 rejections. The estimated power is \(6{,}382/10{,}000=0.6382\), or 63.82%. Its estimated Monte Carlo standard error is

SE_{\text{simulation}} \approx\sqrt{\frac{0.6382(1-0.6382)}{10{,}000}} =\sqrt{\frac{0.6382(0.3618)}{10{,}000}} =\sqrt{0.00002309} \approx0.00481.

The arithmetic is based on \(0.6382(0.3618)=0.23090076\), so the value under the square root is \(0.000023090076\), which rounds to \(0.00002309\). As a rough measure of Monte Carlo precision, 1.96 simulation standard errors are \(1.96(0.00481)\approx0.00942\). A corresponding approximate range is \(0.6382\pm0.00942\), or about 0.6288 to 0.6476. This range describes simulation uncertainty around the estimated power, not uncertainty about the effect in a future study.

What Makes a Simulation Estimate Trustworthy?

A large number of replications can make the estimate numerically stable, but it cannot compensate for a mismatch between the simulated setup and the planned analysis. Check the following before interpreting a result:

The alternative is explicit. Power is tied to a particular alternative model or effect size. A simulation at one value does not provide power at every possible value.
The data model matches the plan. Replicates should reflect the assumed sampling distribution, sample size, independence or dependence structure, and other design features relevant to the test.
The test rule is unchanged. Use the same hypotheses, significance level, test statistic, and rejection threshold for every replicate. Choosing a rule after seeing simulated outcomes changes the procedure being evaluated.
Replications are generated independently. The standard error formula above relies on independent Bernoulli rejection indicators with the same power.
There are enough replications for the needed precision. Report \(R\) and, when helpful, the simulation standard error. A power estimate with only a few replications may be unstable even if the arithmetic is correct.

Simulation is especially useful when the planned procedure is too involved for a simple power formula. The same principle still applies: simulate the data-generating mechanism under a scientifically chosen alternative, then run the entire fixed analysis procedure on each replicate. If a procedure includes multiple steps, all of those steps must be represented consistently in each simulated repetition.

Common Mistake: Changing the Rule or Overreading the Fraction

A common mistake is to simulate data under an alternative, use one rejection threshold in some replications and another threshold in others, and then call the resulting fraction the power of the planned test. That fraction does not estimate the power of one fixed test, because no single decision rule was applied. Set the rule in advance and apply it identically to every generated data set.

Another mistake is to treat a simulated estimate such as 0.738 as exact. It is an estimate based on a finite number of random replications. Report the rejection fraction and the number of replications, and use the simulation standard error to describe its numerical precision. Also keep the model assumptions visible: a precise simulation under an unsuitable alternative model can still answer the wrong planning question.

Key takeaway: Estimated power by simulation is the fraction of independent data sets generated under a specified alternative that reject under one fixed, prespecified test rule. The estimate is only as relevant as its model, and its Monte Carlo standard error quantifies the finite-replication noise.

Check Your Understanding

Try these questions without looking back.

A simulation generates 3,000 independent data sets under a specified alternative, and 2,190 lead to rejection. What is the estimated power?
Why must the significance level and rejection rule be fixed before analyzing the simulated replications?
If an estimated power is 0.60 from \(R=1{,}000\) replications, write the formula for its estimated simulation standard error. What kind of uncertainty does it describe?
A researcher simulates independent Bernoulli outcomes with probability 0.40 but plans to analyze clustered observations. What aspect of the simulation needs reconsideration?
Why does increasing the number of replications improve the numerical precision of the estimated power without correcting a poor choice of alternative model?

What You'll Learn