Uncertainty In Interim Nuisance-Parameter Estimates

Why Interim Estimates Make Sample-Size Updates Uncertain

A sample-size rule is often written as though an interim nuisance-parameter estimate were the true value. For example, a larger estimated outcome variance leads to a larger required sample size, while a lower estimated event rate leads to more participants being needed to observe a target number of events. But an estimate based on limited interim data varies from trial to trial. Consequently, the updated sample size is itself a random quantity, even when the adaptation rule is fixed in advance.

The distinction matters operationally. A rule may have a modest average increase yet a substantial probability of a large increase, or it may often hit its cap. Looking only at the update calculated from one interim estimate does not describe these operating characteristics. This tutorial quantifies that variability under explicit models. It focuses on uncertainty in the nuisance estimate, not on uncertainty in the treatment-effect estimate.

Definition: An interim nuisance-parameter estimate is an estimate, calculated from data available at an interim analysis, of a parameter that affects the sample-size calculation but is not the treatment effect of primary interest. Examples include a common outcome variance and an event probability. The sample-size update distribution is the probability distribution of the sample size produced by the prespecified update rule over repeated interim data sets, under specified model assumptions and parameter values.

Variance Estimates and Sample-Size Variability

Consider a two-arm trial with a continuous outcome. Suppose observations in both arms are independent and normally distributed, with a common variance $\sigma^2$, and the design targets a difference $\delta>0$. For a two-sided normal test at significance level $\alpha$ and target power $1-\beta$, the usual approximate required sample size per arm is

n_{\mathrm{req}}(\sigma^2) = \frac{2\left(z_{1-\alpha/2}+z_{1-\beta}\right)^2\sigma^2}{\delta^2}.

This approximation treats the variance as known and uses the normal distribution for the test statistic. If the interim estimate $s^2$ replaces $\sigma^2$, the continuous-valued updated requirement is proportional to $s^2$. The operational rule may then round up to an integer and apply a floor or cap. To isolate the effect of estimation uncertainty, first consider the uncapped rule without rounding.

Result: If interim data support the model $\nu s^2/\sigma^2\sim\chi^2_\nu$, then for $n_{\mathrm{req}}(s^2)=C s^2$, where $C=2(z_{1-\alpha/2}+z_{1-\beta})^2/\delta^2$, the ratio of the estimated to true-variance sample-size requirements satisfies $$ \frac{n_{\mathrm{req}}(s^2)}{n_{\mathrm{req}}(\sigma^2)} = \frac{s^2}{\sigma^2} \sim \frac{\chi^2_\nu}{\nu}. $$ Its mean is 1 and its variance is $2/\nu$. Thus, its relative standard deviation is $\sqrt{2/\nu}$, before rounding or applying limits.

The chi-square model holds, for example, when $s^2$ is a pooled within-arm variance from independent normal observations with a common variance, using $\nu=n_1+n_2-2$ degrees of freedom for interim arm sizes $n_1$ and $n_2$. It does not automatically hold for every blinded variance estimator: pooling treatment groups can introduce a treatment-effect contribution, and nonnormality or unequal variances can also change the estimator's distribution. The model must match how the interim estimate is actually constructed.

Worked Example: Probability of a Variance-Driven Increase

Suppose a hypothetical equal-allocation trial has a baseline variance estimate $\sigma_0^2=100$ square units, targets $\delta=4$ units, and uses $\alpha=0.05$ with 80% power. Assume the interim variance estimate is based on $\nu=19$ degrees of freedom and follows the chi-square model above. Calculate the uncapped sample-size requirement and the probability that the interim rule calls for an increase of more than 20% relative to the baseline requirement.

Using $z_{0.975}=1.960$ and $z_{0.80}=0.842$, the continuous baseline requirement per arm is

n_0 = \frac{2(1.960+0.842)^2(100)}{4^2} = \frac{2(2.802)^2(100)}{16} = 98.14.

The operational baseline would round up to 99 per arm. Before integer rounding, the ratio of the updated requirement to the baseline is $s^2/100$. An increase of more than 20% occurs when $s^2/100>1.20$, equivalently when $\chi^2_{19}>19(1.20)=22.8$. Therefore,

P\left(\frac{n_{\mathrm{req}}(s^2)}{n_{\mathrm{req}}(100)}>1.20\right) = P(\chi^2_{19}>22.8) = 0.2463.

Under these assumptions, about 24.63% of repeated interim samples would produce an uncapped continuous requirement more than 20% above baseline. The ratio has standard deviation $\sqrt{2/19}=0.3244$, so its variability is appreciable even though its mean is 1. Rounding and any cap would alter the distribution of the final operational sample size.

For numerical reproducibility, retain enough precision until the final rounding. For example, if a normalized calculation defines $K=(z_{0.975}+z_{0.80})^2/0.10^2$, the unrounded normal quantiles give $K\approx784.9$. Using the displayed rounded quantiles instead gives $K=(1.960+0.842)^2/0.10^2=785.1204$, or approximately 785.1. This scalar is not, by itself, a two-arm per-arm sample size; the allocation and design-specific factors in the sample-size formula still apply.

Deriving the Variability Result

The result above follows directly from the sampling distribution of a normal-theory variance estimate. If $Y_1,\ldots,Y_m$ are independent observations from a normal distribution with variance $\sigma^2$, the usual sample variance $s^2$ satisfies $(m-1)s^2/\sigma^2\sim\chi^2_{m-1}$. A pooled within-arm estimate has the analogous result with the residual degrees of freedom $\nu$. Write $Q=\nu s^2/\sigma^2$. Then $Q\sim\chi^2_\nu$, and the uncapped sample-size ratio is $Q/\nu$.

E\left(\frac{Q}{\nu}\right) = \frac{E(Q)}{\nu} = 1, \qquad \operatorname{Var}\left(\frac{Q}{\nu}\right) = \frac{\operatorname{Var}(Q)}{\nu^2} = \frac{2\nu}{\nu^2} = \frac{2}{\nu}.

Here $E(Q)=\nu$ and $\operatorname{Var}(Q)=2\nu$ are the mean and variance of a chi-square random variable with $\nu$ degrees of freedom. This establishes both the unbiasedness of the uncapped variance-based requirement on the continuous scale and its relative variability. It does not imply that the rounded or capped sample size has the same mean: rounding and truncation are nonlinear operations.

Event-Rate Estimates and Required Enrollment

A parallel calculation applies when the design needs a target number $D$ of events and uses an interim estimate of the probability $p$ that an enrolled participant contributes an event by the relevant analysis time. Suppose $m$ independent, fully observed interim participants each have the same event probability $p$, and $X$ is the number with events. Then $X\sim\operatorname{Binomial}(m,p)$, and $\hat p=X/m$. If event probability is stable in the future, a simple enrollment rule is $N_{\mathrm{req}}=D/\hat p$, rounded up. This rule requires $\hat p>0$; in practice a prespecified fallback or cap is needed if $X=0$.

For moderate $m$ and an interior value of $p$, $\hat p$ has approximate variance $p(1-p)/m$. A first-order delta-method approximation, using the derivative of $D/u$ at $u=p$, gives

\operatorname{Var}(N_{\mathrm{req}}) \approx \left(\frac{D}{p^2}\right)^2\frac{p(1-p)}{m} = \left(\frac{D}{p}\right)^2\frac{1-p}{mp}.

This approximation makes the main drivers visible: a smaller pilot, a rarer event, or a larger target increases the variability in required enrollment. There is also a second-order effect. Since $D/u$ is convex for $u>0$, low estimates of $p$ can increase the required enrollment more than equally sized high estimates decrease it. A Taylor expansion gives the approximation $E(D/\hat p)\approx(D/p)\{1+(1-p)/(mp)\}$, when $\hat p$ is sufficiently concentrated away from zero. It is an approximation, not an exact expectation; the reciprocal is problematic at $\hat p=0$, and a cap changes the result.

Worked Example: Uncertainty in an Interim Event Rate

In a hypothetical trial, the planning event probability is $p=0.20$, and the target is $D=60$ events. An interim cohort contains $m=100$ fully observed participants. Under independent Bernoulli outcomes with a stable event probability, the baseline enrollment calculation is $60/0.20=300$ participants. The approximate standard deviation of the interim event-rate estimate is

SE(\hat p) = \sqrt{\frac{0.20(0.80)}{100}} = 0.0400.

Using the delta-method formula, the approximate standard deviation of the continuous enrollment requirement is

SE(N_{\mathrm{req}}) \approx \frac{60}{0.20} \sqrt{\frac{0.80}{100(0.20)}} = 300\sqrt{0.04} = 60.0\text{ participants}.

The second-order approximation to its mean is $300\{1+0.80/[100(0.20)]\}=312$ participants. This modest upward difference illustrates the reciprocal transformation's asymmetry; it should not be interpreted as an exact expected enrollment or used without considering the operational rule.

Suppose the interim data instead show 18 events among the 100 participants. Then $\hat p=0.18$, giving $N_{\mathrm{req}}=60/0.18=333.3$, which rounds up to 334 participants under a simple ceiling rule. An increase greater than 10% over the baseline occurs when $\hat p<0.20/1.10=0.1818$, or $X\le18$. A normal approximation with continuity correction gives $P(X\le18)\approx\Phi((18.5-20)/4)=\Phi(-0.375)\approx0.3538$. Thus, the rule has roughly a 35% chance of calling for an increase above 10% under these assumptions. The approximation can be checked against the exact binomial probability when making an operational decision.

Caps, Diagnostics, and Alternative Planning Choices

Real adaptation rules generally impose bounds. If the continuous requirement is $n^*$, a rule with floor $L$ and cap $U$ might set $n_{\mathrm{final}}=\min\{U,\max\{L,\lceil n^*\rceil\}\}$. For a variance rule, the probability of reaching the cap can be calculated from the corresponding chi-square tail: if $U$ is a per-arm cap and $n^*=n_0s^2/\sigma_0^2$, then the cap is reached when $s^2/\sigma_0^2$ is sufficiently large to require at least $U$ per arm. A floor or cap creates point probability at the bound, so the distribution is no longer the uncapped scaled chi-square distribution. Likewise, a cap limits maximum enrollment but does not eliminate the chance that the desired power is unattainable under the true nuisance value.

Diagnostics should examine the estimate and its data-generating assumptions, not just whether the algorithm ran. For a variance update, inspect arm-specific spreads, outliers, residual patterns, and evidence that the common-variance normal model is implausible. For an event-rate update, check event ascertainment, follow-up maturity, eligibility and calendar-time changes, and whether the interim cohort represents participants expected later. A rate estimated before many participants have complete follow-up can be systematically low, not merely imprecise. In either setting, report the estimate, its uncertainty, the uncapped recommendation, and the final recommendation after applying the rule.

Three planning approaches answer different questions. A plug-in update substitutes the interim estimate directly; it is simple but can obscure how often extreme updates occur. A conservative fixed plan chooses a larger sample size in advance, avoiding interim estimation but potentially enrolling more participants than needed. A stochastic assessment repeatedly generates interim estimates under specified nuisance values, applies the complete rule including caps and rounding, and summarizes probabilities such as reaching the cap or exceeding a prespecified increase. Simulation is especially useful when estimates are bounded, data are incomplete, or the update rule is nonlinear; its conclusions depend on the scenarios and model used.

These calculations have limits. The chi-square and binomial results assume particular sampling models and stable parameters; they do not account automatically for operational changes, missing outcomes, treatment-related variance differences, or dependence among participants. The delta-method event-rate approximations can be poor when $mp$ is small or the estimate may be zero. Finally, quantifying nuisance-estimate variability does not by itself establish the type I error or power of the full adaptive test. Those properties depend on the complete design and analysis procedure.

Common Pitfall

Common mistake: “The interim variance estimate is unbiased, so the adaptive sample size will equal the planned sample size on average, and there is no need to study its variability.” Correction: Unbiasedness applies to the uncapped continuous requirement when it is linear in an unbiased variance estimate. It says nothing about the probability of a large increase, the result after rounding or caps, or the behavior of an inverse event-rate rule. Evaluate the distribution of the actual adaptation rule, not only the mean of its input estimate.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

For a normal-theory variance estimate with $\nu=50$, what are the mean and relative standard deviation of the uncapped sample-size ratio?
Why is the chi-square variance model not automatically valid for every blinded pooled estimate?
In a trial targeting $D$ events, explain why a low interim $\hat p$ increases required enrollment and why the reciprocal rule can have an upward bias.
What changes in the distribution of the final sample size when an upper cap is applied?

Solutions. (1) The mean ratio is 1 and its relative standard deviation is $\sqrt{2/50}=0.2000$, under the stated chi-square model. (2) The model relies on the estimator's construction and assumptions such as independent normal outcomes with a common variance; a blinded pooled estimate may include a treatment-effect contribution, and nonnormality or unequal variances can alter its distribution. (3) Since $N_{\mathrm{req}}=D/\hat p$, lowering $\hat p$ raises the quotient. The reciprocal is convex, so downward deviations can increase the quotient more than comparable upward deviations reduce it; the Taylor approximation quantifies this effect when the estimate is well away from zero. (4) The rule assigns probability to the cap whenever the uncapped recommendation reaches or exceeds it. The final distribution therefore has a point mass at the cap and is not the original uncapped distribution.