Why Software Output Needs an Audit
Power and sample-size software can perform calculations quickly, but it cannot decide whether the inputs represent the study you intend to plan. A report can display several decimal places and still answer the wrong question if the alternative is reversed, a two-sided test is selected instead of a one-sided test, or a standard deviation is entered where the software expects a variance.
The purpose of interpreting output is therefore not just to read off a power or sample size. It is to connect that number to the assumptions and choices that produced it. Before using a result, check the test direction, null and alternative values, variability, significance level, and target power. Then confirm how the software reports sample size and what calculation method it uses.
What to Check in a Power Report
Power software usually needs a statistical model and a set of planning choices. For example, a report for a one-sample test of a mean might require the null mean \(\mu_0\), a particular alternative mean \(\mu_a\), the population standard deviation \(\sigma\), the significance level \(\alpha\), the direction of the alternative, and either the sample size \(n\) or a target power. Interpret each field in context rather than treating it as a label detached from the study question.
| Report item | Question to ask |
|---|---|
| Test direction | Does the software test the intended one-sided or two-sided alternative? |
| Null and alternative values | Is the null value correct, and does the alternative represent the effect worth detecting? |
| Variability | Is the entered quantity a standard deviation, variance, or another model parameter, and are its units correct? |
| Significance level | Does the selected \(\alpha\) match the planned false-positive rate for this test? |
| Power or sample size | Was power calculated for a supplied \(n\), or was \(n\) found for a specified target power? |
| Method and options | Does the calculation use the intended test and approximation, such as a normal approximation or an exact method? |
A one-sided alternative has a direction. For instance, \(H_a:\mu>\mu_0\) asks about an increase, whereas \(H_a:\mu<\mu_0\) asks about a decrease. A two-sided alternative, \(H_a:\mu\ne\mu_0\), counts departures in either direction. Those choices affect the rejection region and hence power. A one-sided calculation is not a harmless shortcut for a two-sided question; it describes a different test.
The variability input also deserves careful inspection. In a normal model \(X\sim N(\mu,\sigma^2)\), \(\sigma\) is the standard deviation and \(\sigma^2\) is the variance. Some software asks for the standard deviation, while other interfaces may ask for the variance. Entering 12 as a variance when the intended standard deviation is 12, for example, changes the implied spread substantially. Units provide a useful check: a standard deviation has the same units as the measurement; a variance has squared units.
Worked Example: Checking a One-Sided Power Result
Suppose a hypothetical laboratory plans a one-sample upper-tail \(Z\) test of a mean measurement. The null value is \(\mu_0=100\) units, the alternative of interest is \(\mu_a=106\) units, the known population standard deviation is \(\sigma=12\) units, \(n=25\), and \(\alpha=0.05\). A software report shows a one-sided test and power \(0.804\). Check whether that value is reasonable and interpret it.
Model and assumptions. The calculation uses independent observations from a population with standard deviation 12 units, and the test statistic is treated as standard normal under \(H_0\). The normal model for the test is appropriate if the measurement distribution is normal, or if the sampling distribution of the mean is adequately approximated by a normal distribution. This is a planning calculation: it treats \(\sigma=12\) as known and evaluates power at the specified alternative mean of 106 units.
For the upper-tail test, the critical value is \(z_{1-\alpha}=z_{0.95}\approx1.6449\). At the alternative mean, the test statistic has a normal distribution with mean shift \((\mu_a-\mu_0)\sqrt{n}/\sigma\). Thus the power is
Substituting the planned values gives
The output \(0.804\) is consistent after rounding. Under this model, if the true mean were 106 units and the study were repeated many times as planned, about 80.4% of those tests would reject \(H_0:\mu=100\) in favor of the upper-tail alternative at \(\alpha=0.05\). This does not say that the probability the alternative is true is 80.4%.
Reading a Sample-Size Result
A sample-size calculation reverses the planning question. Instead of supplying \(n\) and asking for power, the user supplies a target power and asks how large \(n\) must be under the stated model and alternative. Many formulas first produce a real-valued sample-size requirement. Because a study cannot enroll a fraction of an individual, the result must be rounded up to the next whole number. Rounding down can leave the study below its target.
The reason for rounding up is that, for a fixed test, effect, variability, and \(\alpha\), increasing \(n\) generally reduces the standard error and makes the specified alternative easier to distinguish from the null. A noninteger requirement such as 42.03 lies between 42 and 43; choosing 42 does not meet that calculated requirement, while choosing 43 does. A package may display both the unrounded result and the integer result, or it may show only the integer. Check which it reports.
Worked Example: Auditing a Two-Sided Mean Sample Size
A hypothetical study is planned to test a population mean against \(H_0:\mu=50\) units using a two-sided \(Z\) test. The team wants 90% power to detect a difference of 5 units in either direction, assumes a known population standard deviation of 10 units, and sets \(\alpha=0.05\). Software reports a required sample size of 43 and achieved power of about 0.906. Verify the sample size and interpret the report.
Model and assumptions. This is a two-sided one-sample \(Z\) test with independent observations, known \(\sigma=10\) units, and a normal test statistic under the null. The planning alternative is \(\mu=55\) units (or, by symmetry for this two-sided test, \(\mu=45\) units). The model assumes the standard deviation and alternative difference are appropriate planning values.
For a two-sided test with \(\alpha=0.05\), the critical normal value is \(z_{1-\alpha/2}=z_{0.975}\approx1.960\). For target power 0.90, \(z_{0.90}\approx1.2816\). The usual normal-approximation planning calculation is
Substituting the stated values,
Rounding up gives \(n=43\). At \(n=43\), the standardized shift under the alternative is \((55-50)\sqrt{43}/10\approx3.2787\). The two-sided normal power is approximately
The reported result is therefore plausible: 43 observations per study sample are required under these assumptions, and the achieved power at that integer sample size is about 90.6%, slightly above the 90% target. The small excess is a normal consequence of rounding up. It is not evidence that the software has changed the target power.
Checking Proportion Inputs and Approximation
For a proportion, a software report may ask for a null proportion \(p_0\), an alternative proportion \(p_a\), a direction, sample size, and significance level. The proportion values must be between 0 and 1. Check that the direction agrees with their ordering: a claim that the proportion increases from the null value requires an upper-tail alternative. Also check whether the package uses a normal approximation, a continuity correction, or an exact binomial calculation, since these methods need not return identical values.
Worked Example: Interpreting Power for a One-Sample Proportion
Consider a hypothetical quality-monitoring plan. A test will assess whether the fraction of items meeting a specified criterion exceeds \(p_0=0.10\). The planning alternative is \(p_a=0.18\), the planned sample size is \(n=200\), and \(\alpha=0.05\). A normal-approximation power report gives approximately 0.952. Check the direction, conditions, and numerical interpretation.
Model and assumptions. Each observation is modeled as an independent Bernoulli outcome with the same success probability. The calculation uses the one-sample normal-approximation \(Z\) test for \(H_0:p=0.10\) against \(H_a:p>0.10\). The expected counts under the null are \(np_0=20\) successes and \(n(1-p_0)=180\) failures; under the alternative they are \(np_a=36\) successes and \(n(1-p_a)=164\) failures. These counts support a normal approximation for this illustrative planning calculation. A package using a different method can produce a somewhat different result.
Under the null, the standard error used for the test statistic is \(\sqrt{p_0(1-p_0)/n}\), which here is
The upper-tail rejection boundary for the sample proportion is approximately
Under \(p_a=0.18\), the standard deviation of the sample proportion is approximately \(\sqrt{p_a(1-p_a)/n}=\sqrt{0.18(0.82)/200}\approx0.02717\). The boundary is about \((0.1349-0.18)/0.02717\approx-1.660\) standard deviations below the alternative mean. Therefore,
The output's direction matches the question, and its value is consistent with the stated normal approximation. In repeated studies following this model, about 95.2% would reject the null when the true proportion is 0.18. The result is conditional on the assumed null and alternative proportions, sample size, test, and significance level; it is not a guarantee for any one study.
A Practical Audit of Output
A reliable reading follows a consistent order. This prevents a plausible-looking number from distracting from a mismatched input. Read the report in context, and compare it with the written study plan rather than relying only on the software's defaults.
Write \(H_0\) and \(H_a\) in words and symbols. Confirm whether the alternative is one-sided or two-sided and, if one-sided, whether it points in the intended direction.
Check that the stated null value and the alternative effect are scientifically meaningful, use the correct units, and are not accidentally interchanged.
Verify whether the software expects a standard deviation, variance, or proportion, and confirm the assumptions that justify the selected test and approximation.
Make sure the significance level is the intended value. If the software asks for target power, distinguish that requested threshold from the power calculated at the final sample size.
For sample size, confirm rounding up and whether the report gives per-group or total enrollment when groups are involved. Note any exact, approximate, or continuity-corrected method.
Software packages can differ even when the displayed inputs appear similar. One package might use a normal approximation, another an exact calculation, and another a correction at the rejection boundary. Some interfaces ask for the effect size directly, while others ask for both null and alternative values. When outputs disagree, compare all settings and methods before deciding that one calculation is wrong.
Common Mistake: Treating the Output as Independent of Its Inputs
A common error is to read “power = 0.80” as a general property of a study design, without mentioning the alternative, test direction, \(\alpha\), or variability. That reasoning is incomplete: power is conditional on those choices. A calculation with a larger difference from the null can have greater power than one with a smaller difference, even at the same sample size. Likewise, changing the direction or \(\alpha\) changes the rejection rule and can change power.
The correction is to report the result together with its key conditions. For example: “With \(n=25\), a one-sided test at \(\alpha=0.05\), known \(\sigma=12\) units, and true mean 106 units rather than the null mean of 100 units, the calculated power is about 0.804 under the normal model.” That statement explains what the number answers and makes its limitations visible.
Check Your Understanding
Use the stated model and inputs to decide what each output means.
- A report uses \(H_a:\mu<\mu_0\), but the study question asks whether the mean is greater than \(\mu_0\). What must be checked or changed before using its power value?
- Why should you confirm whether an interface requests \(\sigma\) or \(\sigma^2\) before entering a variability value?
- A target-power calculation returns \(n^*=58.2\). What integer sample size should be planned, and why?
- In a one-sided test of \(p>p_0\), what does a power of 0.90 mean when the specified alternative proportion is true?
- Name two reasons different software packages might report different power values for apparently similar inputs.