Power Calculations With Unknown Variance

Why Unknown Variance Changes the Calculation

For a normal endpoint, a trial’s observed treatment-control difference is uncertain not only because the future sample means vary, but also because the endpoint variance must be learned from the trial. A Bayesian decision uses the posterior distribution after observing those data. Conditional Bayesian power is therefore the probability, under a specified true data-generating scenario, that the resulting posterior decision will meet the trial’s success criterion.

With known variance, the posterior probability can often be evaluated directly from a normal posterior for the treatment effect. When variance is unknown, the posterior for each arm’s mean is heavier-tailed, and the success probability generally depends on both the observed arm means and their within-arm variation. Simulation provides a practical way to evaluate that probability and then estimate how often the decision succeeds across future trials.

Definition: Let $\mu_C$ and $\mu_T$ be the true control- and treatment-arm means, and let $\sigma_0^2$ be the true common endpoint variance used to generate future data. For an analysis posterior and a rule that declares success when $P(\theta>\delta\mid D)\geq\gamma$, conditional Bayesian power is $$ \operatorname{Power}(\mu_{C0},\mu_{T0},\sigma_0^2) = P_{\mu_{C0},\mu_{T0},\sigma_0^2} \left\{ P(\theta>\delta\mid D)\geq\gamma \right\}, $$ where $\theta=\mu_T-\mu_C$, $D$ denotes the future trial data, $\delta$ is the effect threshold, and $\gamma$ is the required posterior probability.

The two true means are specified separately, not only through their difference. If the analysis prior assigns separate informative distributions to the arm means, shifting both true means while holding their difference fixed can change the distribution of the posterior decision. The analysis prior, sample sizes, success rule, and any allocation or design features must also be held fixed when reporting this power.

Analysis Model and Posterior Updates

Consider an independent-arm model with an unknown mean and variance in each arm. For arm $j\in\{C,T\}$, the observed outcomes are conditionally independent and normally distributed. Use an independent normal-inverse-gamma prior in each arm:

Y_{ij}\mid\mu_j,\sigma_j^2\sim N(\mu_j,\sigma_j^2), \qquad \sigma_j^2\sim\operatorname{InvGamma}(a_{0j},b_{0j}), \qquad \mu_j\mid\sigma_j^2\sim N\left(m_{0j},\frac{\sigma_j^2}{k_{0j}}\right).

Here the inverse-gamma density is proportional to $x^{-(a+1)}\exp(-b/x)$ for $x>0$. The prior parameters satisfy $a_{0j},b_{0j},k_{0j}>0$. The arms’ prior distributions are independent. This is an analysis model: it describes how data will be analyzed, and it need not be identical to the fixed data-generating scenario used to calculate operating characteristics.

For $n_j$ observations in arm $j$, let $\bar{y}_j$ be the sample mean and $SS_j=\sum_i(y_{ij}-\bar{y}_j)^2$ the within-arm sum of squares. The posterior has the same form, with parameters

k_{nj}=k_{0j}+n_j,\qquad m_{nj}=\frac{k_{0j}m_{0j}+n_j\bar{y}_j}{k_{nj}}, \qquad a_{nj}=a_{0j}+\frac{n_j}{2},

b_{nj}=b_{0j}+\frac{SS_j}{2} +\frac{k_{0j}n_j}{2k_{nj}}(\bar{y}_j-m_{0j})^2.

These updates follow by multiplying the normal likelihood by the prior and completing the square in $\mu_j$. In particular, the identity

\sum_i(y_{ij}-\mu_j)^2+\frac{k_{0j}}{1}(\mu_j-m_{0j})^2 = SS_j+k_{nj}(\mu_j-m_{nj})^2 +\frac{k_{0j}n_j}{k_{nj}}(\bar{y}_j-m_{0j})^2

separates the terms involving the updated mean from the remaining sum-of-squares contribution. The remaining terms update $a_{0j}$ and $b_{0j}$, yielding the expressions above. Conditional on $\sigma_j^2$, the posterior mean is normal with mean $m_{nj}$ and variance $\sigma_j^2/k_{nj}$; integrating over the posterior variance gives a scaled Student $t$ marginal distribution for $\mu_j$.

Because the two arm posteriors are independent under this model, posterior draws of $\theta$ can be constructed by drawing each arm’s mean from its marginal posterior and subtracting. Unless a convenient analytic calculation is available, the posterior probability $P(\theta>\delta\mid D)$ is estimated by the fraction of these draws exceeding $\delta$.

Worked Example: Updating the Analysis Prior

Suppose a hypothetical trial has $n_C=n_T=10$ patients, control and treatment sample means $\bar{y}_C=0.8$ and $\bar{y}_T=2.4$ endpoint units, and within-arm sums of squares $SS_C=20$ and $SS_T=18$ endpoint-units squared. In each arm, use $m_0=0$, $k_0=0.01$, $a_0=2$, and $b_0=2$. Take $\delta=0$ endpoint units and $\gamma=0.95$.

For either arm, $k_n=0.01+10=10.01$ and $a_n=2+10/2=7$. For control,

m_{nC}=\frac{0.01(0)+10(0.8)}{10.01}=0.7992,

and

b_{nC}=2+\frac{20}{2} +\frac{0.01(10)}{2(10.01)}(0.8-0)^2 =12.0032.

For treatment,

m_{nT}=\frac{0.01(0)+10(2.4)}{10.01}=2.3976, \qquad b_{nT}=2+\frac{18}{2} +\frac{0.01(10)}{2(10.01)}(2.4)^2 =11.0288.

The posterior mean difference is $2.3976-0.7992=1.5984$ endpoint units. The posterior probability of a positive difference is not determined by this mean alone: it also depends on posterior uncertainty in both arm means. Suppose $100{,}000$ independent pairs of posterior draws produce $98{,}140$ differences above zero. The estimated probability is $98{,}140/100{,}000=0.9814$. Since $0.9814\geq0.95$, this hypothetical dataset meets the specified posterior decision rule. This is a posterior calculation for one dataset, not a power estimate.

Simulating Conditional Power

To estimate power, repeat the trial under fixed true values. For each outer replicate, generate $n_C$ outcomes from $N(\mu_{C0},\sigma_0^2)$ and $n_T$ outcomes from $N(\mu_{T0},\sigma_0^2)$. Apply the stated analysis prior and posterior decision rule to that replicate. The success indicator is one if the estimated posterior probability meets $\gamma$, and zero otherwise. The average of these indicators estimates conditional power.

Specify the scenario and rule.
Fix both true arm means, the true variance, sample sizes, the analysis prior, $\delta$, and $\gamma$. State units and the allocation.

Generate future trial data.
For each replicate, independently generate normal outcomes in each arm under the specified true means and variance. Check that the data-generating assumptions match the intended scenario.

Apply the analysis model.
Calculate each arm’s sufficient summaries and posterior parameters. Draw posterior values of $\mu_C$ and $\mu_T$, then estimate $P(\theta>\delta\mid D)$.

Record and average decisions.
Record success if the posterior probability is at least $\gamma$. Repeat for $R$ future trials and average the $R$ success indicators.

Estimator: If $I_r=1$ when replicate $r$ meets the posterior success rule and $I_r=0$ otherwise, then $$ \widehat{\operatorname{Power}}=\frac{1}{R}\sum_{r=1}^R I_r. $$ Ignoring inner posterior-simulation error, its Monte Carlo standard error is estimated by $$ \widehat{SE}_{MC} = \sqrt{\frac{\widehat{\operatorname{Power}}\left(1-\widehat{\operatorname{Power}}\right)}{R}}. $$

Worked Example: One Simulated Future Trial

Consider a hypothetical design with 20 patients per arm. The fixed data-generating values are $\mu_{C0}=0$, $\mu_{T0}=1$ endpoint unit, and $\sigma_0^2=4$ endpoint-units squared. Use the independent analysis prior $m_0=0$, $k_0=1$, $a_0=2$, and $b_0=2$ in both arms. Let $\delta=0.2$ endpoint units and $\gamma=0.90$.

Suppose one generated dataset has $\bar{y}_C=0.7$, $\bar{y}_T=1.6$, $SS_C=72$, and $SS_T=76$ endpoint-units squared. Each arm has $k_n=21$ and $a_n=12$. The posterior means are $m_{nC}=20(0.7)/21=0.6667$ and $m_{nT}=20(1.6)/21=1.5238$. The posterior variance parameters are

b_{nC}=2+36+\frac{20}{42}(0.7)^2=38.2333, \qquad b_{nT}=2+38+\frac{20}{42}(1.6)^2=41.2190.

Draw $100{,}000$ posterior mean pairs from the two arm posteriors. Suppose $91{,}480$ draws have $\mu_T-\mu_C>0.2$. The estimated posterior probability is $91{,}480/100{,}000=0.9148$, which exceeds $0.90$, so this replicate is a success. Its outcome is one success indicator; it does not by itself establish that the design has high power.

Worked Example: Estimating Power and Its Monte Carlo Error

For the same hypothetical design and success rule, suppose a simulation of $R=5{,}000$ future trials at $\mu_{C0}=0$, $\mu_{T0}=1$, and $\sigma_0^2=4$ records 3,670 successes. Then

\widehat{\operatorname{Power}}=\frac{3670}{5000}=0.7340, \qquad \widehat{SE}_{MC} =\sqrt{\frac{0.7340(1-0.7340)}{5000}} =\sqrt{0.0000390488} =0.006249.

The estimated conditional power is 73.40%, with an outer Monte Carlo standard error of about 0.625 percentage points. This error measures simulation variability from using a finite number of generated trials; it is not uncertainty about the fixed true means or variance. If the posterior probability in each replicate is itself estimated with finite posterior draws, that inner numerical error can also affect whether the replicate is classified as a success.

Diagnostics, Alternatives, and Limitations

A reliable calculation requires the future-data model and analysis procedure to be implemented consistently. Verify that the simulated arm sizes, means, and variances match the chosen scenario over many replicates; check posterior updates against hand calculations for a small dataset; and ensure that posterior draws use the stated prior and variance parameterization. Record random-number seeds and the inner and outer simulation counts so the result can be reproduced.

The decision is discontinuous at $\gamma$: a small numerical error in the posterior probability can change a replicate from failure to success. Increase the number of posterior draws when many replicates lie near the threshold, or use a more accurate posterior-probability calculation where feasible. Repeat the power calculation with more outer replicates to assess Monte Carlo stability. A reported estimate should include the outer Monte Carlo standard error, and it should not imply precision finer than that error supports.

This arm-specific model permits different unknown variances in the two arms. A pooled common-variance analysis is an alternative, but it imposes a different analysis model and requires corresponding posterior updates. The earlier material on Normal-Endpoint Power With Known Variance treats the case where variance is fixed and known; it cannot simply be reused by substituting an estimated variance while retaining the same posterior calculation.

Conditional power is evaluated at the fixed data-generating values. Averaging power over a prior distribution for uncertain true effects or nuisance parameters changes the target to assurance, rather than merely adding simulation replicates. Also, a normal model may be unsuitable for strongly skewed, bounded, or heavy-tailed outcomes; informative analysis priors can make decisions sensitive to the absolute outcome location as well as the treatment-control contrast. These assumptions and sensitivities should be examined in scenarios relevant to the planned trial.

Common pitfall: Defining power only at a treatment effect and variance, then simulating both arm means, leaves the operating characteristic incompletely specified when the analysis prior is location-informative. Specify $\mu_{C0}$ and $\mu_{T0}$ separately, or justify a location-invariance condition for the analysis rule and prior. A second error is to treat $P(\theta>\delta\mid D)$ as the power: it is a posterior probability for one observed dataset, whereas power is the repeated-trial probability that this posterior probability reaches $\gamma$.

Check Your Understanding

Answer each question; concise solutions are provided after the questions.

For an analysis prior that gives separate informative distributions to $\mu_C$ and $\mu_T$, why might power depend on the two true means, not just their difference?
A simulation has 2,000 future trials and 1,510 meet the posterior success criterion. Calculate estimated power and its outer Monte Carlo standard error.
What is the difference between estimating $P(\theta>\delta\mid D)$ for one generated dataset and estimating conditional power?
If the estimated posterior probability is close to $\gamma$, what computational diagnostic is especially important?

Solutions. (1) The prior can assign different support across absolute locations; shifting both true means changes how likely the generated data are under those prior distributions and can alter posterior decisions. (2) The estimate is $1510/2000=0.755$. Its standard error is $\sqrt{0.755(0.245)/2000}=\sqrt{0.0000924875}=0.009617$, or about 0.962 percentage points. (3) The first is a within-dataset posterior probability; conditional power is the repeated-data probability that this posterior probability is at least $\gamma$, under fixed true data-generating values. (4) Assess inner Monte Carlo error and increase posterior draws or use a more accurate calculation, because small error can flip the success indicator.

Power Calculations With Unknown Variance

What You'll Learn