Conditional Power with Estimated Variance

Why Estimated Variance Changes Conditional Power

For a mean comparison, conditional power asks how likely the remaining observations are to produce a successful final analysis, given the data available at an interim analysis and a specified effect for the future. When the outcome variance is known, the information at each look is determined. In practice, the variance is usually estimated, so the interim estimate is often substituted into the information calculation. That substitution makes the calculation usable, but it does not make the estimated variance known or remove its sampling uncertainty.

This tutorial develops the plug-in normal approximation for two independent, equally allocated groups with a continuous outcome. It focuses on one final look after the interim analysis. The calculation can be extended to several remaining looks, but then conditional power is the probability of crossing at least one of the remaining design boundaries, not merely a one-look tail probability.

Model and assumptions: At each look, observations in the two groups are independent, with normally distributed outcomes, a common variance \(\sigma^2\), and a mean difference \(\delta=\mu_T-\mu_C\). Allocation is equal, and the observations added after the interim analysis are independent of the interim data. The normal approximation treats the variance estimate observed at interim as a fixed plug-in value for calculating future information. The resulting conditional power is approximate, not an exact conditional probability for a studentized final test.

Plug-In Information and the Conditional Distribution

Suppose there are \(n\) observations per group at a given look. If the common variance were known, the variance of the difference in sample means would be \(2\sigma^2/n\). The information for the mean difference is therefore \(I=n/(2\sigma^2)\). With an estimated common variance, use the interim pooled sample variance \(s_p^2\) in its place, giving plug-in information \(\hat I=n/(2s_p^2)\). Here \(s_p^2\) is the pooled within-group variance estimate, not the variance of the observed group means.

Let \(\hat I_1\) be the plug-in information at interim and \(\hat I_F\) the plug-in information planned at the final look, with \(\hat I_F>\hat I_1\). For the calculation below, carry the interim variance estimate forward, so both information values use the same \(s_p^2\). Let \(\hat\delta_1\) be the interim difference in sample means and define the interim standardized statistic \(Z_1=\sqrt{\hat I_1}\hat\delta_1\). Conditional power requires an assumed future effect, denoted \(\delta^*\). This could be a design effect or another effect chosen for the calculation; it is not automatically the interim estimate.

Under the working normal model with variance fixed at its plug-in value, the final standardized statistic has an approximate conditional normal distribution. Its conditional mean combines the contribution already observed with the expected contribution from the remaining information.

Z_F\mid \text{interim data}\ \approx\ N\left( \sqrt{\frac{\hat I_1}{\hat I_F}}Z_1+ \frac{\hat I_F-\hat I_1}{\sqrt{\hat I_F}}\delta^*, \ 1-\frac{\hat I_1}{\hat I_F} \right).

Write \(m\) for the conditional mean and \(v=1-\hat I_1/\hat I_F\) for the conditional variance. If success at the final look requires \(Z_F\ge c\), where \(c\) is the boundary specified by the group sequential design, the plug-in conditional power is

\widehat{\mathrm{CP}}= P(Z_F\ge c\mid\text{interim data},\delta^*) \approx 1-\Phi\left(\frac{c-m}{\sqrt v}\right).

The boundary \(c\) must be the applicable design boundary, not automatically the ordinary single-look normal critical value. Group sequential designs often use boundaries calibrated across looks to control the overall type I error rate. Using an inappropriate boundary changes the success probability being calculated.

Why the Formula Has This Form

A short derivation makes clear which part of the calculation comes from the interim data and which part is uncertain. On the information scale, represent the standardized-score numerator at information \(I\) as \(S(I)=\delta^*I+W(I)\), where \(W(I)\) has independent normal increments and an increment over an information increase \(I_F-I_1\) has variance \(I_F-I_1\). This is the working normal score-process approximation. Given the interim score \(S(I_1)=\sqrt{I_1}Z_1\), the future increment is independent of that score.

Z_F=\frac{S(I_F)}{\sqrt{I_F}} =\sqrt{\frac{I_1}{I_F}}Z_1+ \frac{(I_F-I_1)\delta^*}{\sqrt{I_F}}+ \frac{W(I_F)-W(I_1)}{\sqrt{I_F}}.

The last term has conditional mean zero and variance \((I_F-I_1)/I_F\). Substituting the interim plug-in information values for \(I_1\) and \(I_F\) gives the stated approximation. This argument conditions on the observed interim score and uses the future independent-increment model. It does not account exactly for the fact that the final test statistic may divide by a newly estimated variance.

Worked Calculations

Worked Example: Conditional Power With a Plug-In Pooled Variance

Consider a hypothetical equally allocated trial with 20 observations per group at interim and 40 per group at the final look. The interim pooled standard deviation is \(s_p=10\) outcome units, and the observed difference in means is 4 outcome units. Suppose conditional power is evaluated under \(\delta^*=5\) outcome units, with an illustrative upper boundary \(c=1.96\).

The interim and final plug-in information values are

\hat I_1=\frac{20}{2(10^2)}=0.1000,\qquad \hat I_F=\frac{40}{2(10^2)}=0.2000.

Thus \(Z_1=\sqrt{0.1000}(4)=1.2649\), and \(\hat I_1/\hat I_F=0.5\). The conditional mean is

m=\sqrt{0.5}(1.2649)+ \frac{0.2000-0.1000}{\sqrt{0.2000}}(5) =0.8944+1.1180=2.0124.

The conditional variance is \(1-0.5=0.5\), so the standardized distance from the conditional mean to the boundary is

\frac{1.96-2.0124}{\sqrt{0.5}}=-0.0741.

Therefore \(\widehat{\mathrm{CP}}\approx1-\Phi(-0.0741)=0.5296\), or about 53%. Under the assumed future effect and this illustrative boundary, the normal approximation gives roughly a 53% chance of exceeding the final boundary. It is not a probability that the assumed effect is true.

Worked Example: A Small Interim Sample

In a second hypothetical trial, there are 6 observations per group at interim and 12 per group at the final look. Suppose \(s_p=12\) outcome units, the observed interim difference is 6 outcome units, the assumed future effect is \(\delta^*=4\) outcome units, and the illustrative final boundary is again \(c=1.96\).

The information values and interim statistic are

\hat I_1=\frac{6}{2(12^2)}=0.020833,\qquad \hat I_F=\frac{12}{2(12^2)}=0.041667,\qquad Z_1=\sqrt{0.020833}(6)=0.8660.

The information ratio is \(0.5\). Substituting into the conditional mean gives

m=\sqrt{0.5}(0.8660)+ \frac{0.041667-0.020833}{\sqrt{0.041667}}(4) =0.6124+0.4082=1.0206.

The conditional variance is \(0.5\), and

\frac{1.96-1.0206}{\sqrt{0.5}}=1.3289,\qquad \widehat{\mathrm{CP}}\approx1-\Phi(1.3289)=0.092.

The estimated conditional power is about 9.2%. The arithmetic is valid for the stated plug-in calculation, but the small interim sample is a warning: its pooled variance estimate may be imprecise. Treating that estimate as fixed can make the reported probability look more certain than the variance information warrants.

Worked Example: Sensitivity to the Interim Variance Estimate

Suppose a hypothetical trial has 10 observations per group at interim and 20 per group at the final look. The observed interim mean difference is 3 outcome units, the assumed future effect is 4 outcome units, and the illustrative boundary is \(c=1.96\). First take the interim pooled standard deviation to be 8 outcome units.

Then

\hat I_1=\frac{10}{2(8^2)}=0.078125,\qquad \hat I_F=\frac{20}{2(8^2)}=0.15625,\qquad Z_1=\sqrt{0.078125}(3)=0.8385.

The information ratio is again \(0.5\). Hence

m=\sqrt{0.5}(0.8385)+ \frac{0.15625-0.078125}{\sqrt{0.15625}}(4) =0.5929+0.7906=1.3835.

The boundary distance is \((1.96-1.3835)/\sqrt{0.5}=0.8152\), so the conditional power is approximately \(1-\Phi(0.8152)=0.2075\), or about 20.7%.

Now keep the sample sizes and observed mean difference fixed but use \(s_p=10\) outcome units. The information values become \(0.0500\) and \(0.1000\), and \(Z_1=\sqrt{0.0500}(3)=0.6708\). The conditional mean is

m=\sqrt{0.5}(0.6708)+ \frac{0.1000-0.0500}{\sqrt{0.1000}}(4) =0.4743+0.6325=1.1068.

Thus \((1.96-1.1068)/\sqrt{0.5}=1.2066\), giving conditional power of approximately \(1-\Phi(1.2066)=0.114\), or 11.4%. The change does not establish that one variance is correct; it shows that estimated variance can materially affect conditional power.

Small-Sample Assessment and Practical Checks

The key approximation is not simply that the outcome is normal. It is also that the interim variance estimate can stand in for the unknown variance when describing the future standardized statistic. With few observations, \(s_p^2\) can vary substantially from sample to sample. Under the equal-variance normal model, the pooled variance estimate is independent of the group mean difference at a given look, but it is still random. Plugging in its observed value does not eliminate that randomness or reproduce the exact distribution of a final statistic studentized by a variance estimate that includes future observations.

For a small sample, the exact final test may use a \(t\) statistic and a \(t\) critical value rather than a normal statistic and a normal boundary. Even then, a standard \(t\) critical value alone does not automatically give the exact conditional power after observing interim data: the final denominator includes variance information from both the observed and future data. An exact conditional calculation must account for that denominator and the conditioning on the interim results. If exact calculation is impractical, a simulation that generates the future observations under the stated model, retains the observed interim data, and applies the planned final analysis can assess the normal approximation.

Useful checks include verifying that \(\hat I_F>\hat I_1\), that \(0<1-\hat I_1/\hat I_F<1\), and that the information units are consistent with the outcome units used for \(\delta^*\). If \(\hat I_F=\hat I_1\), there is no future information and the remaining conditional variance is zero; the formula for a future normal tail is then not applicable as written. Also check the assumed effect and boundary explicitly. A conditional power calculated under \(\delta^*\) answers a different question from one calculated under another effect.

The common-variance assumption matters. If the groups have materially different variances, a pooled estimate and information \(n/(2s_p^2)\) may not represent the variance of the mean difference well; use an analysis and variance model consistent with the trial design. A variance estimate that changes sharply between looks, outliers, non-normal outcomes with very small samples, or discrepancies between the planned and observed allocation are also reasons to examine the approximation rather than report a single plug-in number without qualification.

Approach	What it uses	Main qualification
Known-variance normal calculation	A specified variance and normal score increments	Usually a reference calculation; the variance is rarely known in practice.
Plug-in normal calculation	The observed interim variance estimate carried forward into information	Simple and transparent, but it does not fully represent variance-estimation uncertainty.
Studentized or simulation-based calculation	The planned final test, including its variance estimator and boundary	Can better reflect small-sample behavior, but requires more computation and careful conditioning.

A further distinction is the effect assumed for the future. Conditional power under a prespecified design effect asks what would happen if future data followed that effect. Substituting the interim effect estimate instead produces a different plug-in calculation and can be particularly unstable when the interim sample is small. Neither calculation averages over uncertainty about the effect; that is a different probability assessment.

Common Mistake

Pitfall: Treating \(s_p\) as though it were the known population standard deviation and calling the resulting conditional power an exact probability. Correction: Label the result as a plug-in normal approximation, state the assumed future effect and design boundary, and assess sensitivity to plausible variance values or use a calculation that simulates the planned studentized analysis. The observed interim estimate supplies a working scale; it does not remove uncertainty about \(\sigma^2\).

Check Your Understanding

Use the plug-in normal approximation, and state any assumptions needed. Answers follow the questions.

With 8 observations per group at interim, 16 per group at the final look, and \(s_p=4\) outcome units, calculate \(\hat I_1\) and \(\hat I_F\).
For the information values in Question 1, suppose \(Z_1=0.8\) and \(\delta^*=2\) outcome units. Using \(c=1.96\), calculate the conditional mean and conditional power.
Why is carrying the interim pooled variance estimate forward not an exact treatment of variance uncertainty?
What should replace \(c=1.96\) if the actual group sequential design specifies a different final boundary?

Solutions. (1) \(\hat I_1=8/[2(4^2)]=0.2500\) and \(\hat I_F=16/[2(4^2)]=0.5000\), in reciprocal squared outcome units. (2) The ratio is \(0.5\), so

m=\sqrt{0.5}(0.8)+\frac{0.5000-0.2500}{\sqrt{0.5000}}(2) =0.5657+0.7071=1.2728.

The conditional variance is \(0.5\), and \((1.96-1.2728)/\sqrt{0.5}=0.9719\). Thus the approximate conditional power is \(1-\Phi(0.9719)\approx0.166\), or 16.6%. (3) The interim pooled variance is random, and the final studentized statistic also uses future variance data; fixing the interim estimate does not reproduce that full conditional distribution. (4) Use the boundary specified by the group sequential design, because it defines the success event being evaluated.