From Planned Power to Achieved Power
A power calculation is conditional on a model and a set of design inputs. If the true variability is larger than anticipated, observations are dependent, or fewer subjects contribute data than planned, the study can have less power than its design promised. The practical question is not only “What was the planned power?” but also “Under the conditions that actually occurred, what rejection probability did the design have?”
Here, planned power means the rejection probability calculated under the assumptions chosen before data collection. Achieved power means the rejection probability under the conditions that generated the data, using a specified test and its rejection rule. It is a property of the assumed alternative, test, and design; it is not a probability that the null hypothesis is false.
A useful diagnosis separates four possible gaps: the effect may differ from the planned effect; the variance may differ; the sampling distribution may not match the approximation; or the effective sample size may be smaller than the enrolled sample size. These are distinct issues. For example, increasing variance changes the standard error, while dependence changes the variance formula itself.
A Normal-Model Benchmark
Use the following benchmark to make the role of assumptions explicit. Suppose \(X_1,\ldots,X_n\) are independent observations from a normal population with mean \(\mu\) and known standard deviation \(\sigma\). For a one-sided test of \(H_0:\mu=\mu_0\) against \(H_1:\mu>\mu_0\), reject when the standardized statistic exceeds the upper standard-normal critical value \(z_{1-\alpha}\), where \(P(Z\le z_{1-\alpha})=1-\alpha\) for \(Z\sim N(0,1)\).
Under a true mean \(\mu=\mu_0+\Delta\), with \(\Delta>0\), the statistic has a normal distribution with variance 1 and mean \(\Delta\sqrt n/\sigma\). Therefore the power is
This expression is exact under the stated normal, independence, and known-variance assumptions. In other settings it may be an approximation, or it may not apply. In particular, replacing \(\sigma\) by a planning value that is not the actual standard deviation does not make the formula describe achieved power.
Worked Example: A Larger-Than-Planned Standard Deviation
A hypothetical monitoring study plans a one-sided test with \(\alpha=0.05\), \(n=64\), and a target mean increase \(\Delta=2\) units. The planning standard deviation is 8 units. Assume a normal model, independent observations, and a known standard deviation for the test. Find planned power, then recalculate power if the actual standard deviation is 10 units.
Method. Use the one-sided normal power formula. The critical value is \(z_{0.95}=1.6449\).
Under the planning assumptions, the standardized mean shift is
Thus planned power is
Now retain the same \(n\) and effect but use the actual standard deviation, \(\sigma=10\). The shift is \(2(8)/10=1.6\), so achieved power under this corrected variance model is
The increase in standard deviation from 8 to 10 reduces power from about 0.6388 to 0.4821, a drop of about 0.1567. This calculation assumes the test is calibrated using the correct standard deviation. Common mistake: retaining the planning value 8 in the power formula after learning that the true standard deviation is 10. That repeats the planned calculation; it does not assess achieved power. The recalculation uses the actual standard deviation throughout.
Dependence Changes the Variance
A sample size count alone does not determine precision. If observations are dependent, the variance of an average or contrast depends on their covariance as well as on their individual variances. For paired measurements, a positive correlation can improve precision compared with independent measurements, while a lower-than-planned correlation can remove some of that gain.
For paired observations \((X_j,Y_j)\), suppose each marginal standard deviation is \(\sigma\), and the correlation within each pair is \(\rho\). The difference \(D_j=X_j-Y_j\) has variance
If the pairs are independent and \(D_j\) is normal with known standard deviation \(\sigma_D\), a one-sided test of a mean difference has the same normal-model power formula, with \(n\) equal to the number of pairs and \(\sigma\) replaced by \(\sigma_D\). It is essential not to treat the two measurements within a pair as independent when calculating \(\sigma_D\).
Worked Example: Less Within-Pair Correlation Than Planned
A hypothetical matched study plans \(n=40\) pairs to detect a mean difference of \(\Delta=4\) units. Each measurement has standard deviation 10 units. The planned within-pair correlation is \(\rho=0.50\), but the correlation is later estimated as \(\rho=0.20\). Assume independent pairs, normal pair differences, a one-sided test at \(\alpha=0.05\), and known difference standard deviation for each calculation. Compare planned and recalculated power.
Method. First calculate the standard deviation of a pair difference from \(2\sigma^2(1-\rho)\), then use the one-sided normal power formula. With the planned correlation,
The planned standardized shift is \(4\sqrt{40}/10=2.5298\). Since \(z_{0.95}=1.6449\), planned power is
Using the actual correlation instead gives
The recalculated shift is \(4\sqrt{40}/12.6491=2.0000\), so achieved power is
The lower correlation makes pair differences more variable and reduces power. Common mistake: using the marginal standard deviation 10 as if it were the standard deviation of the difference under both correlations. The relevant quantity is the pair-difference standard deviation, which must include the covariance term. If the analysis continues to use a variance assumption that is wrong, its actual type I error can also differ from the nominal level; power comparisons here assume correct calibration at each variance.
Distributional Shape and Small Samples
Normal approximations can be useful, but their accuracy depends on the sampling distribution and sample size. For small samples of Bernoulli observations, the sum is discrete and can be skewed. If the test’s rejection rule is stated in terms of an integer count, exact binomial probabilities give the rejection probability directly. A normal approximation may differ because it smooths the discrete distribution.
Worked Example: Exact Power for a Discrete Count Rule
A hypothetical screening study observes \(X\), the number of positive responses among 10 independent participants. It uses the rule “reject \(H_0:p=0.50\) if \(X\ge 8\)” against a larger response probability. Find the rule’s actual significance level and its exact power when \(p=0.80\). Then compare the power with a normal approximation using a continuity correction.
Method. Under a response probability \(p\), \(X\sim\operatorname{Binomial}(10,p)\). The actual significance level is the probability of the rejection event under the null:
At \(p=0.80\), the exact power is
For comparison, under \(p=0.80\), the binomial mean is \(10(0.8)=8\) and the variance is \(10(0.8)(0.2)=1.6\). With continuity correction, \(X\ge8\) is approximated by a normal variable exceeding 7.5. The resulting approximation is
The exact power exceeds the approximation by about \(0.6777995-0.6536836=0.02412\). Common mistake: treating the approximation as exact, or using the boundary 8 without a continuity correction. The discrete count rule determines exact power here; the approximation is only a diagnostic comparison. Also note that this rule’s actual significance level is about 0.0547, not exactly 0.05.
Lost Observations and the Analyzable Sample Size
Power depends on the number of observations that contribute to the analysis, not merely on the number initially recruited. Attrition therefore needs to be translated into an expected analyzable sample size. If observations are missing completely at random and the remaining observations retain the planned distribution and independence, a first approximation is to replace the planned \(n\) by the expected number observed. If missingness is related to outcomes or treatment response, that substitution alone may not describe the resulting analysis; assumptions about the missing-data process require separate attention.
Worked Example: Allowing for Expected Attrition
A hypothetical study needs 100 complete observations to test a one-sided mean increase of 3 units with known standard deviation 10, significance level 0.05, and independent normal observations. Investigators expect 20% of enrolled participants not to provide usable data. Compare power if they enroll only 100 people with power if they recruit enough to expect 100 usable observations. Assume losses are independent of outcomes and the complete observations meet the stated model.
Method. First distinguish enrollment from the expected analyzable sample. If only 100 are enrolled, the expected usable count is \(100(0.80)=80\). For \(n=80\), the standardized shift is \(3\sqrt{80}/10=2.6833\), giving
If the target is 100 usable observations, divide by the retention proportion: \(100/0.80=125\). Recruiting 125 people gives an expected 100 usable observations under the stated attrition model. At \(n=100\), the shift is \(3\sqrt{100}/10=3\), so power is
Thus enrolling 100 rather than 125 produces an expected analyzable count of only 80 and lowers the corresponding power from about 0.9123 to 0.8505. These are calculations based on expected sample sizes; if the final count is random, power can vary with that realized count. Common mistake: inserting the enrollment count directly into the power formula despite anticipated losses. The relevant \(n\) is the number of usable, independent observations under the analysis model.
Choosing Sensitivity Analyses
A sensitivity analysis asks how the design’s conclusions change when uncertain planning inputs take plausible alternative values. It does not repair a misspecified model, but it reveals which assumptions are consequential and whether the design is robust to uncertainty. A useful analysis changes one input at a time to diagnose its effect, then considers plausible combinations when several departures may occur together.
Identify the target effect, standard deviation, correlation or clustering, distributional shape, significance level, and expected retention rate relevant to the design.
Use defensible low, central, and high values for uncertain quantities. Keep fixed design choices, such as the rejection rule and planned allocation, visible.
Use the correct variance and sampling distribution for each scenario. Use exact probabilities when the model and rejection region make them available, rather than treating an approximation as exact.
Report which assumption drives the largest change and whether the result depends on treating observations as independent or complete. If the missingness mechanism or test calibration changes, address that explicitly.
For a mean comparison, the standard deviation and target effect are often influential because the normal-model shift is \(\Delta\sqrt n/\sigma\). In paired or clustered designs, the correlation structure can be just as important. For a small-sample count test, the exact discrete rejection probability may be sensitive to the cutoff and the assumed success probability. Retention rates matter when the design requires a particular number of analyzable observations. These inputs should be checked rather than buried in a single headline power value.
Finally, achieved power based on an estimated effect from the same study can be misleading as a summary of evidence: the estimate and its uncertainty already describe what the data say. For design assessment, the more informative practice is to state the actual test and assumptions, then show how power changes across plausible effects, variances, dependence levels, and usable sample sizes. A single recalculated value is meaningful only when its conditions are explicit.
Check Your Understanding
For each problem, state or use the assumptions given and check that the sample size in the calculation is the analyzable sample size.
- A one-sided normal test has \(\alpha=0.05\), \(n=36\), known \(\sigma=3\), and target increase \(\Delta=1.5\). Find its power. (Answer: 0.9123)
- For the design in question 1, suppose the actual standard deviation is 4.5, with all other conditions unchanged and the test calibrated using the correct standard deviation. Find the recalculated power. (Answer: 0.6388)
- In a paired normal design, there are 25 independent pairs, each measurement has standard deviation 8, the actual within-pair correlation is 0.75, and the target mean difference is 3. For a one-sided test at \(\alpha=0.05\) with known difference standard deviation, find power. (Answer: approximately 0.8430)
- A count \(X\) follows \(\operatorname{Binomial}(8,0.75)\), and a test rejects when \(X\ge6\). Find the exact rejection probability at this alternative. (Answer: 0.6785)
- A design needs 90 usable observations and expects 10% loss, independent of outcomes. How many people should be recruited, rounded up, to reach the target expected usable count? (Answer: 100)