Bias From Sample-Size Re-Estimation

Why Sample-Size Re-Estimation Can Affect Estimation

A sample-size re-estimation rule is often designed to improve power or to reach a planned precision. But if the rule uses the interim treatment-effect estimate, it also determines how much that estimate contributes to the final pooled estimate. The resulting estimator need not be unbiased, even when both stage-specific estimates are unbiased before adaptation.

The key issue is not simply that the total sample size changes. It is that the weight assigned to the interim estimate can depend on the value of that estimate. For example, if a large interim effect leads to a smaller second stage, the interim data receive a larger share of the final estimate precisely when they show a large effect. This association can pull the pooled estimate away from the true effect.

Two-Stage Normal Model

Definition: Let \(\theta\) be the true treatment effect. In a two-stage trial, let \(X\) be the stage-one estimate based on \(n_1\) observations, and let \(Y\) be the stage-two estimate based on \(n_2(X)\) new observations. Assume \(X\sim N(\theta,\sigma^2/n_1)\), and, conditional on the stage-one data, \(Y\sim N(\theta,\sigma^2/n_2(X))\). The stage-two observations are independent of stage one, and the rule \(n_2(X)\) is fixed in advance. The common known \(\sigma^2\) is the variance scale for the treatment-effect estimate per observation. The pooled estimator is \(\hat{\theta}=\{n_1X+n_2(X)Y\}/\{n_1+n_2(X)\}\).

This model assumes unbiased normal stage-specific estimates, the same treatment-effect estimand and variance scale in both stages, and no unmodelled dependence between the new stage-two data and the interim data. It also assumes the only adaptation considered here is the choice of \(n_2\) from \(X\). These assumptions make the effect of an estimate-dependent weight transparent; real trials may require a different variance model.

Write \(a(X)=n_1/\{n_1+n_2(X)\}\), the weight on \(X\). Then \(\hat{\theta}=a(X)X+\{1-a(X)\}Y\). If \(n_2\) is fixed, the weight is fixed, and the pooled estimator is unbiased. When \(n_2\) depends on \(X\), the weight \(a(X)\) is random and associated with the interim estimation error.

Bias and Variance: Derivation

Given the stage-one data, \(X\) and \(a(X)\) are known, while the conditional expectation of \(Y\) is \(\theta\). Therefore

E(\hat{\theta}\mid X)=a(X)X+\{1-a(X)\}\theta =\theta+a(X)(X-\theta).

Averaging over stage one gives the exact bias:

\operatorname{Bias}(\hat{\theta}) =E(\hat{\theta})-\theta =E\{a(X)(X-\theta)\}.

Thus the stage-specific estimates can each be unbiased and the pooled estimate can still be biased. The issue is the covariance between the interim error \(X-\theta\) and its weight. If high values of \(X\) receive more weight, positive interim errors tend to be emphasized more than negative ones.

The law of total variance gives a corresponding decomposition. Conditional on \(X\), only the stage-two estimate is random, so \(\operatorname{Var}(\hat{\theta}\mid X)=\{1-a(X)\}^2\sigma^2/n_2(X)\). The conditional mean varies with \(X\), giving

\operatorname{Var}(\hat{\theta}) =E\left[\{1-a(X)\}^2\frac{\sigma^2}{n_2(X)}\right] +\operatorname{Var}\{a(X)(X-\theta)\}.

The first term is the average remaining stage-two variance after weighting. The second is variation in the conditional mean induced by stage one. Neither term alone is a general measure of the estimator’s performance: bias and variance are distinct, and mean squared error is variance plus squared bias.

For a differentiable weight function \(a\) with a bounded derivative and suitable vanishing boundary terms, the bias has a useful normal-model identity. Let \(v=\sigma^2/n_1\) and let \(f\) be the density of \(X\). Since \(f'(x)=-(x-\theta)f(x)/v\), integration by parts yields

E\{a(X)(X-\theta)\} =-v\int a(x)f'(x)\,dx =v\int a'(x)f(x)\,dx =vE\{a'(X)\}.

So, under these conditions, an increasing weight function produces nonnegative bias, and a decreasing one produces nonpositive bias. A threshold rule is not differentiable at its threshold, so this identity should not be applied to it without accounting for the jump. The examples below calculate threshold-rule bias directly.

Worked Example: A Threshold Rule at a Zero True Effect

Hypothetical setup. Suppose \(n_1=100\), \(\sigma=10\), and the true effect is \(\theta=0\) units. Thus \(X\sim N(0,1)\). The rule sets \(n_2=100\) when \(X\leq 0\), but \(n_2=50\) when \(X>0\). The stage-two observations are new and independent, with variance \(100/n_2\) for \(Y\).

When \(X\leq 0\), \(a=100/200=1/2\); when \(X>0\), \(a=100/150=2/3\). Since \(E\{(X-\theta)I(X>0)\}=1/\sqrt{2\pi}\), the bias is

\operatorname{Bias}(\hat{\theta}) =\left(\frac{2}{3}-\frac{1}{2}\right)\frac{1}{\sqrt{2\pi}} =\frac{1}{6\sqrt{2\pi}} =0.06649\text{ units}.

The bias is positive: positive stage-one estimates receive more weight because they trigger the smaller second stage. The stage-specific estimates remain unbiased; it is the data-dependent weighting that changes the pooled estimate’s expectation.

The variance can also be computed explicitly. For \(U=X-\theta\sim N(0,1)\), symmetry gives \(E\{U^2I(U>0)\}=E\{U^2I(U\leq0)\}=1/2\). The contribution from the varying conditional mean to the second moment is therefore \((1/2)^2(1/2)+(2/3)^2(1/2)=25/72\). The expected conditional stage-two variance is \((1/2)(1/2)^2(1)+(1/2)(1/3)^2(2)=17/72\).

Consequently, \(E\{(\hat{\theta}-\theta)^2\}=25/72+17/72=7/12=0.5833\) squared units. Subtracting the squared bias gives \(\operatorname{Var}(\hat{\theta})=7/12-\{1/(6\sqrt{2\pi})\}^2=0.5789\) squared units. For comparison, a fixed \(n_2=100\) gives an unbiased pooled estimate with variance \(0.5000\) squared units. This comparison is specific to the stated rule and model; it is not a claim that every re-estimation rule increases variance.

Worked Example: A Threshold Different From the True Effect

Hypothetical setup. Keep \(n_1=100\) and \(\sigma=10\), but let the true effect be \(\theta=1\) unit. Use the same rule: \(n_2=100\) for \(X\leq0\) and \(n_2=50\) for \(X>0\). Now \(U=X-\theta\sim N(0,1)\), and \(X>0\) is equivalent to \(U>-1\).

For a standard normal variable, \(E\{U I(U>c)\}=\phi(c)\), where \(\phi\) is the standard normal density. At \(c=-1\), this expectation is \(\phi(1)=0.24197\). The weight jumps by \(2/3-1/2=1/6\) above the cutoff, so

\operatorname{Bias}(\hat{\theta}) =\frac{1}{6}\phi(1) =\frac{1}{6}(0.24197) =0.04033\text{ units}.

For the variance, \(P(U>-1)=0.84134\) and \(E\{U^2I(U>-1)\}=\Phi(1)-\phi(1)=0.59922\), where \(\Phi\) is the standard normal distribution function. The complementary second moment is \(1-0.59922=0.40078\). The varying conditional mean contributes \((1/2)^2(0.40078)+(2/3)^2(0.59922)=0.36652\) to the second moment. The expected conditional variance is \((0.15866)(1/2)^2(1)+(0.84134)(1/3)^2(2)=0.22663\).

Thus the mean squared error is \(0.36652+0.22663=0.59315\) squared units. Subtract the squared bias, \(0.04033^2=0.001627\), to obtain a variance of \(0.5915\) squared units. The cutoff’s location relative to \(\theta\) changes how often each weight is used, so the bias differs from the zero-effect example.

When a Size Change Does Not Create This Bias

A sample-size change by itself does not imply bias. If the size is selected independently of \(X\), then conditional on that selection the weights are fixed and the stage-specific estimates have expectation \(\theta\). Averaging over the selection still gives \(E(\hat{\theta})=\theta\). This independence condition matters: merely calling a rule “blinded” does not establish that its selected size is independent of the treatment-effect estimate under every model.

Worked Example: An Independent Random Sample Size

Hypothetical setup. Let \(n_1=50\), \(\sigma=10\), so \(\operatorname{Var}(X)=2\). Suppose an external random choice, independent of all treatment-effect data, sets \(n_2=50\) with probability \(1/2\) and \(n_2=150\) with probability \(1/2\). Both \(X\) and \(Y\) are unbiased for \(\theta\).

For either selected size, the conditional expectation of the pooled estimate is \(\theta\), so its unconditional bias is \(0\). If \(n_2=50\), the weights are \(1/2\) and \(1/2\), and the variance is \((1/2)^2(2)+(1/2)^2(2)=1.000\) squared units.

If \(n_2=150\), the weights are \(1/4\) and \(3/4\), while \(\operatorname{Var}(Y)=100/150=2/3\). The variance is \((1/4)^2(2)+(3/4)^2(2/3)=0.125+0.375=0.500\) squared units. Since the independent selection has equal probabilities, the unconditional variance is \((1/2)(1.000)+(1/2)(0.500)=0.750\) squared units. Random size selection changes the variance here, but not the expectation. The contrast with effect-dependent selection is the dependence of the weight on the interim estimate.

Diagnostics, Alternatives, and Limitations

In an actual design, inspect the rule as an estimator rule, not only as a power calculation. For each possible interim estimate, record the selected \(n_2\) and the resulting weight \(a(X)\). A plot of interim effect against weight often reveals whether unusually high or low estimates receive systematically greater influence. Under the stated normal model, evaluate \(E\{a(X)(X-\theta)\}\) and the variance decomposition over a clinically relevant range of \(\theta\), using numerical integration or simulation when the rule is complex. Simulations should apply the exact re-estimation rule, including bounds and rounding, and should report bias, variance, and mean squared error separately.

A fixed second-stage size avoids this particular effect-dependent weighting. A size rule based only on information independent of the treatment-effect estimate avoids this bias under the independence and unbiasedness assumptions, though it can still change variance. Reporting the two stage-specific estimates alongside the pooled estimate can make the source of the final weighting visible. Conditional-error calibration, covered in earlier material on conditional error, concerns error control for a test; it does not by itself establish that a pooled point estimate is unbiased.

The calculations here have limitations. They use a common known variance scale, normal stage estimates, a fixed estimand, and a rule based on one interim estimate. Unknown variances, multiple treatment arms, changed estimands, additional adaptation variables, or dependence between stages can alter both formulas and conclusions. The direction and size of bias are properties of the joint design and estimation rule, not universal consequences of re-estimation.

Common mistake: “Each stage estimate is unbiased, so any weighted average is unbiased.” That reasoning is valid for fixed weights, but not for weights chosen from the stage-one estimate. When \(a(X)\) depends on \(X\), the relevant quantity is \(E\{a(X)(X-\theta)\}\), which need not be zero.

Check Your Understanding

Try each question, then compare your reasoning with the solutions below.

In the two-stage model, write the bias of the pooled estimator in terms of \(a(X)\).
If \(n_2\) is fixed in advance and both stage estimates are unbiased, what is the bias of the pooled estimator?
In the zero-effect worked example, why is the bias positive?
Suppose a sample-size choice is independent of \(X\), but takes two possible values. Does that alone make the pooled estimator biased under the stated model?

Solutions. (1) The bias is \(E\{a(X)(X-\theta)\}\), obtained by conditioning on \(X\) and then averaging. (2) The weight \(a\) is fixed, so \(E\{aX+(1-a)Y\}=a\theta+(1-a)\theta=\theta\); the bias is zero. (3) Positive \(X\) triggers the smaller second stage and thus the larger stage-one weight, \(2/3\) rather than \(1/2\). Positive interim errors are consequently given greater influence, yielding bias \(1/(6\sqrt{2\pi})>0\). (4) No. Conditional on either independently selected size, the weights are fixed and the pooled estimator has expectation \(\theta\); averaging those conditional expectations still gives \(\theta\). Its variance can nevertheless differ from that under a fixed size.