Sensitivity Analysis Across Plausible Effects

Why Examine More Than One Effect?

A sample-size plan often needs an assumed effect: the difference a study is intended to detect. But the true effect is not known when the design is chosen, and a single assumed value can make the planned power seem more certain than it is. If the effect is smaller than assumed, the same study may have substantially less power; if it is larger, power may be much higher.

Sensitivity analysis makes this dependence visible. Rather than calculating power for only one alternative, calculate it for a range of plausible alternatives while holding the other design choices fixed. The result helps show which conclusions are robust to the assumed effect and which depend strongly on it. The range should be justified by the scientific context and practical importance, not chosen just because it produces a preferred design conclusion.

Power at a particular alternative is the probability that the test rejects \(H_0\) when that alternative is true. Calculating power across alternatives does not assign probabilities to those alternatives. For example, reporting power at effects of 1, 2, and 3 units does not say that each effect is equally likely, or that the true effect must be one of those three values.

Model and Power Formulas

We will use a one-sample z test for a population mean as a clear setting for the calculations. Suppose independent observations come from a normal population with known standard deviation \(\sigma\), and the study has a fixed sample size \(n\). Then \(\bar{x}\) has a normal sampling distribution with mean \(\mu\) and standard deviation \(\sigma/\sqrt{n}\). Define the standard error \(SE=\sigma/\sqrt{n}\). The test compares \(H_0:\mu=\mu_0\) with an alternative about the mean.

These assumptions give an exact normal model when the population is normal. A z-based approximation can also be used under suitable conditions for a sufficiently large sample, but the examples below use the stated normal model. Let \(\delta=\mu-\mu_0\) denote the true difference from the null value. Under that alternative, the standardized test statistic has a normal distribution with mean \(\delta/SE\) and variance 1.

For an upper-tailed test with significance level \(\alpha\), reject \(H_0\) when \(Z>z_{1-\alpha}\), where \(z_{1-\alpha}\) is the standard normal quantile with cumulative probability \(1-\alpha\). Thus the power at effect \(\delta\) is:

Here \(\Phi\) is the cumulative distribution function of a standard normal random variable. This formula makes the sensitivity question explicit: with the design fixed, power changes as \(\delta/SE\) changes. In the upper-tailed setting, larger positive effects produce larger power.

For a two-sided test, reject when \(Z<-z_{1-\alpha/2}\) or \(Z>z_{1-\alpha/2}\). The alternative can be in either direction, so power includes the probability of falling in either rejection tail:

For a two-sided test, power depends on the magnitude of the effect: positive and negative effects with the same absolute size have the same power under this symmetric model. This does not mean that the direction of an effect is unimportant scientifically; it means the two-sided rejection rule treats both directions symmetrically.

Why the Formula Gives Power

The power expression follows directly from the rejection region and the distribution of \(Z\) under the alternative. Under a true difference \(\delta\), write \(Z\sim N(\delta/SE,1)\). For the upper-tailed test, standardizing the event \(Z>z_{1-\alpha}\) gives:

The standardized quantity \((Z-\delta/SE)\) has a standard normal distribution, which justifies the final probability. For the two-sided test, the rejection events are disjoint, so their probabilities add. The lower-tail probability is \(\Phi(-z_{1-\alpha/2}-\delta/SE)\); the upper-tail probability is \(1-\Phi(z_{1-\alpha/2}-\delta/SE)\). Adding them gives the two-sided formula above.

One-Sided Sensitivity Across Plausible Effects

A useful sensitivity display may list the assumed effects beside their powers or show the values on a power curve. In either form, keep the test direction, \(\alpha\), sample size, variability assumption, and other design features fixed while varying the effect. Otherwise it becomes difficult to tell which change accounts for a change in power.

Two-Sided Sensitivity and Direction

Comparing Designs Over the Same Effect Range

Sensitivity analysis can also show whether a design comparison depends on which effect is considered plausible. In the next example, the significance level, direction, and assumed variability are held fixed; only sample size differs between the designs. This is a comparison of planned power, not a claim that a larger study is always the best choice regardless of cost, feasibility, or scientific value.

How to Use a Sensitivity Analysis

The calculations are most informative when the effect range is meaningful. Choose a set of values that represents defensible scenarios, such as a smallest effect worth detecting, a central planning value, and larger effects that remain plausible. State the units and whether each value is a difference from the null. If the range is not well justified, a detailed table of powers can create a false sense of precision.

A sensitivity analysis is not a substitute for choosing a scientifically meaningful target. It makes the consequences of that choice transparent. In particular, it can reveal that a design is adequate only if the true effect is relatively large, or that it remains reasonably effective across a broad range of relevant effects.

Common Mistake: Treating Scenarios as Probabilities

A common mistake is to calculate power at several effects and then describe the largest power as the study's overall power. For example, if power is 0.85 at a 3-unit effect and 0.17 at a 1-unit effect, it is not correct to say the study has 85% power without stating the assumed 3-unit effect. Power is conditional on a particular true parameter value and the specified model and design.

A related mistake is to average the powers across a few scenarios and call the result the probability that the study will detect an effect. That average has no general probability interpretation unless a justified probability distribution over true effects is specified and the averaging is explicitly defined. Sensitivity analysis ordinarily reports the conditional powers separately; it does not supply such a distribution.

Finally, rounding should be consistent. If a critical value is rounded before substitution, the resulting power should be reported as the result of that rounded calculation. Alternatively, retain the unrounded critical value throughout. Do not combine tail probabilities after rounding each term so coarsely that their sum changes at the reported precision.