The Pitfall of Observed Post Hoc Power

What Observed Post Hoc Power Calculates

After a hypothesis test, it can seem helpful to calculate the study’s “observed power”: the probability that the test would reject the null if the true effect were equal to the effect estimated from the data. The calculation is possible, but it is usually a poor way to learn what the data say. In common settings, observed power is largely a restatement of the test statistic and p-value, not new evidence about the effect.

To make this precise, consider a one-sample test of a population mean. Suppose observations are independent and normally distributed with known standard deviation \(\sigma\), and the sample size \(n\) is fixed. We test \(H_0:\mu=0\) against \(H_a:\mu\ne 0\) using a two-sided z test at significance level \(\alpha\). Under this model, the standard error is \(SE=\sigma/\sqrt{n}\), and the test rejects when the absolute value of the standardized statistic exceeds the appropriate normal critical value.

Observed power is therefore conditional on a particular model, test, significance level, sample size, and assumed effect. It is not, in general, a probability or forecast for a future study. A future study’s rejection probability equals this calculation only if its relevant design and assumptions match and the true effect is assumed to equal the observed estimate.

Why It Largely Restates the Test Result

For the two-sided z test above, write the observed estimate as \(\hat{\theta}\), its standard error as \(SE\), and the observed test statistic as \(z_{\text{obs}}=\hat{\theta}/SE\). The null is zero, so the estimate is also the observed difference from the null value. Let \(\Phi\) denote the cumulative distribution function of a standard normal random variable, and let \(c=z_{1-\alpha/2}\) be the positive critical value. If the true effect were \(\hat{\theta}\), the test statistic would have a normal distribution with mean \(\hat{\theta}/SE\) and standard deviation 1. The observed power is consequently:

The two terms are the probabilities of crossing the positive and negative rejection boundaries under the plug-in effect. By contrast, the two-sided p-value is \(2\Phi(-|z_{\text{obs}}|)\). Both quantities are determined by the same absolute observed z statistic when the test and standard error are fixed.

The monotonic relationship can be checked directly. For a positive plug-in standardized effect \(d=|z_{\text{obs}}|\), power is \(g(d)=\Phi(d-c)+\Phi(-d-c)\). Its derivative is \(g'(d)=\phi(d-c)-\phi(d+c)\), where \(\phi\) is the standard normal density. Since \(d,c>0\) imply \(|d-c|<d+c\), and the standard normal density decreases as the absolute value of its argument increases, \(\phi(d-c)>\phi(d+c)\). Therefore \(g'(d)>0\): larger absolute z statistics produce larger observed power. The p-value moves in the opposite direction because \(2\Phi(-d)\) decreases as \(d\) increases.

This exact relationship applies to the stated z-test setup. Other tests or calculations may not yield this exact formula, especially if their standard errors or degrees of freedom are handled differently. The central concern remains: inserting the observed estimate as though it were the true effect uses the same data twice, first to obtain the estimate and then to calculate a power value from it.

Use the Confidence Interval to Assess Uncertainty

A confidence interval reports estimated effects and their uncertainty in the units of the problem. In repeated samples under the model, a 95% confidence-interval procedure covers the fixed population parameter in 95% of samples. Once an interval has been calculated, it is not correct to say there is a 95% probability that the fixed parameter lies in that particular interval. Instead, examine the interval’s center, width, and range: which effect sizes are consistent with the data at the interval’s confidence level, and which would be substantively important?

For the known-\(\sigma\) one-sample mean model, the 95% interval is \(\bar{x}\pm1.96SE\). Other settings require the interval procedure appropriate to their model; the point here is to use an interval that matches the analysis, not to apply this formula universally. A confidence interval cannot remove uncertainty, but it expresses it in a way that makes the magnitude and precision of effects visible.

Plan Power Around an Effect of Interest

Power is valuable when used before data collection to evaluate a proposed design. A researcher specifies a meaningful effect size in advance, along with a model, test, significance level, and planned sample size. The resulting power answers a design question: if the specified effect were true and the assumptions held, how often would this planned test reject the null in repeated studies?

This use differs from observed post hoc power in a crucial way. The target effect is chosen for scientific or practical reasons, rather than selected because it happened to be the estimate in the completed study. Planning across a range of plausible effects can also show how conclusions depend on the assumed effect size. Power is not a guarantee that a particular study will detect an effect, nor does it measure the probability that the effect is true.

Common Pitfall and Practical Guidance

Wrong reasoning: “The observed power is low, so the study shows there is probably no effect,” or “the observed power is high, so a future study is likely to be significant.” Neither conclusion follows. Observed power assumes the observed estimate is the true effect; it does not assign probabilities to possible true effects. It is not, in general, an unconditional forecast for a future study. Only when the future test’s relevant design and assumptions match, and its true effect is assumed to equal the observed estimate, does its rejection probability equal the plug-in observed power.

Correction: Report the estimate and a suitable confidence interval to communicate effect size and uncertainty. Interpret the test result using the p-value and prespecified decision rule, without treating either as a measure of the probability that the null is true. If the question is whether a design can reliably detect an effect that matters, calculate power prospectively for that specified effect and design. The earlier material on why a nonsignificant result does not prove no effect addresses that related interpretive issue; a low observed-power value does not resolve it.

In practice, observed power may be requested by software or included in a report, but its presence does not make it informative beyond the test result in the fixed-standard-error z setting developed here. The more useful questions are: What effect did the study estimate? How precise is that estimate? What range of effects is consistent with the data? And, for a future study, what power would its design have for a scientifically meaningful target effect?