Why an Unbiased Stage Estimate May Not Give an Unbiased Final Estimate
In a two-stage trial, the estimate from each stage may be unbiased for the same treatment effect, yet the final estimate can still be biased. The key issue is not simply that the design changes after an interim analysis. It is that the interim result can help determine how much influence that same result has on the final estimate. If an unusually high interim estimate receives more weight, the final estimate tends to be pulled upward; if it receives less weight, it can be pulled downward.
This tutorial isolates that mechanism. It considers estimation of one common treatment effect, not choosing among treatments. The calculations concern bias across repeated trials under a specified true effect. Bias is an average property of an estimator; it does not tell us the error in any particular trial.
The conditional-unbiasedness assumption for stage 2 matters. It says that, once the interim information and the chosen design are fixed, the expected stage 2 estimate is still \(\theta\). It holds in the simple normal-outcome setting when future observations are independent of stage 1 outcomes and are analyzed with a valid stage 2 estimator. It need not hold if further outcome-dependent stopping, missingness, or a misspecified analysis changes the distribution of the stage 2 estimate.
The Source of Bias
Taking expectations of the weighted estimate shows precisely where bias can enter. Since \(W\) is determined before stage 2 outcomes are observed, it is part of \(\mathcal{F}_1\). Therefore, conditional unbiasedness of \(\hat{\theta}_2\) gives \(\mathrm{E}[(1-W)\hat{\theta}_2]=\mathrm{E}[(1-W)\theta]\).
Because \(\mathrm{E}(\hat{\theta}_1-\theta)=0\), the final expression is also \(\mathrm{Cov}(W,\hat{\theta}_1)\). Thus, in this model, the source of bias is the association between the stage 1 estimation error and the weight assigned to stage 1. If \(W\) is fixed, or is random but independent of the stage 1 estimate, this term is zero. If the adaptation makes \(W\) depend on the interim result, it is generally nonzero.
This identity is about the estimator’s unconditional expectation across repetitions of the trial. It does not say that every adaptive estimator is biased, or that a nonzero bias must be large. Nor does it say that the inferential procedure used for a test has an invalid error rate. Estimation bias and calibration of a hypothesis test are different properties.
A Normal Threshold Rule
A useful case makes the dependence visible. Suppose \(\hat{\theta}_1\sim N(\theta,s_1^2)\), where \(s_1\) is the stage 1 standard error. Set \(W=w_H\) when \(\hat{\theta}_1>c\), and \(W=w_L\) otherwise. Here \(c\) is a prespecified threshold and \(w_H,w_L\) are fixed weights. Define \(\phi\) and \(\Phi\) as the standard normal density and cumulative distribution function.
Let \(Z=(\hat{\theta}_1-\theta)/s_1\) and \(a=(c-\theta)/s_1\). Since \(\mathrm{E}(\hat{\theta}_1-\theta)=0\), only the weight difference contributes to the bias:
For the last equality, \(\phi'(z)=-z\phi(z)\), so \(\int_a^\infty z\phi(z)\,dz=[-\phi(z)]_a^\infty=\phi(a)\). The density \(\phi(a)\) is positive. Therefore, if \(w_H>w_L\), the bias is positive; if \(w_H<w_L\), it is negative. Its magnitude is largest when the threshold is near the true mean, because that is where \(\phi((c-\theta)/s_1)\) is largest. This is a derivation for this particular threshold rule, not a claim that every adaptation rule has this form.
Worked Example: Giving More Weight to a Positive Interim Estimate
Suppose a hypothetical two-stage trial estimates a treatment effect in outcome units. Under the true effect \(\theta=0\), let the stage 1 estimate have standard error \(s_1=1\) unit. The rule assigns weight \(w_H=0.8\) to stage 1 if \(\hat{\theta}_1>0\), and \(w_L=0.2\) otherwise. Assume the stage 2 estimate satisfies the conditional-unbiasedness assumption.
Here \(c=0\), \(a=(0-0)/1=0\), and the weight difference is \(0.8-0.2=0.6\). Substituting into the derived expression,
The final estimator has expected value \(0+0.2393=0.2393\) outcome units under this rule and model. The bias is upward because positive interim errors receive more weight than negative interim errors. This calculation is an expectation over repeated trials; it is not a prediction that a particular trial’s estimate will be exactly 0.2393 units above the true effect.
Worked Example: Reversing the Weight Rule
Now suppose a hypothetical trial has true effect \(\theta=0.5\) outcome units and stage 1 standard error \(s_1=1\) unit. The rule gives stage 1 weight \(w_H=0.2\) if \(\hat{\theta}_1>0\), and \(w_L=0.8\) otherwise. Stage 2 again has conditional expectation \(\theta\).
The threshold is \(c=0\), so \(a=(0-0.5)/1=-0.5\). Since \(\phi(-0.5)=0.3521\), the weight difference is \(0.2-0.8=-0.6\), and
The expected final estimate is \(0.5-0.2113=0.2887\) outcome units. The sign is negative because the rule gives less stage 1 weight when the estimate is above the threshold than when it is below. The result illustrates that the direction of bias follows the relationship between the interim error and its weight, not simply whether the true effect is positive or negative.
Data-Dependent Precision and Pooled Estimates
The same mechanism can arise without a rule explicitly described as “weighting the interim result.” For a common-variance normal outcome, the ordinary pooled estimate weights each stage mean by its sample size. If the stage 2 sample size is chosen from the interim result, the pooled weight on stage 1 also depends on that result. The example below illustrates the algebra; it does not require that this particular adaptation be desirable.
Worked Example: A Pooled Estimate with an Interim-Dependent Sample Size
Consider a hypothetical trial with one observation in stage 1 and a known outcome standard deviation of 1 unit. The true treatment effect is \(\theta=0\), so the stage 1 estimate \(\hat{\theta}_1\sim N(0,1)\). Choose one stage 2 observation if \(\hat{\theta}_1>0\), and three if \(\hat{\theta}_1\leq 0\). Assume future observations are independent of stage 1 and have mean \(\theta\). The pooled estimate weights each stage mean by its sample size.
When \(\hat{\theta}_1>0\), the stage 1 weight is \(1/(1+1)=0.5\). When \(\hat{\theta}_1\leq 0\), it is \(1/(1+3)=0.25\). Thus \(w_H=0.5\), \(w_L=0.25\), and \(a=0\). The normal threshold calculation gives
Although the stage 2 mean is conditionally unbiased, the pooled estimate has positive bias under this rule: the interim estimate receives more weight when it is positive. The calculation isolates the weight-dependence mechanism. It does not imply that every sample-size adaptation produces bias of this sign or size; the answer depends on the adaptation rule and the estimator.
What to Check, and What the Result Does Not Say
A useful first diagnostic is to write the estimator as a function of the interim information and then ask whether the stage 1 error affects its own coefficient. In the two-stage weighted form, examine \(\mathrm{E}\{W(\hat{\theta}_1-\theta)\}\). For an actual design, that expectation can be evaluated analytically when the model and rule permit, or approximated by simulation under specified values of \(\theta\). Simulations should reproduce the adaptation rule, generate independent future outcomes under the chosen design, and compare the average final estimate with the true \(\theta\). A simulation is a diagnostic for the specified model, not a proof that the model is correct.
A practical comparison helps keep the issue in focus:
| Weighting situation | Bias under the stated model | Reason |
|---|---|---|
| Fixed \(W\) | Zero | \(\mathrm{E}\{W(\hat{\theta}_1-\theta)\}=W\mathrm{E}(\hat{\theta}_1-\theta)=0\). |
| Random \(W\) independent of \(\hat{\theta}_1\) | Zero | The weight does not favor positive or negative stage 1 errors on average. |
| \(W\) chosen using interim data | Not generally zero | The expectation of the product of weight and interim error may be nonzero. |
The comparison is conditional on the assumptions already stated. In particular, if the future stage estimate is not conditionally unbiased, the displayed bias identity no longer describes the whole problem: additional terms can arise from stage 2. Also, a zero value of this bias term does not establish that an interval has correct coverage, that a test has the intended type I error, or that the estimator is efficient.
The derivation has further limits. It treats one common treatment effect, a two-stage estimator that is a weighted average, and a rule decided from interim information. More complicated estimators may not have this form, and adaptation can affect their distributions in other ways. The identity diagnoses one source of bias; it is not a general correction formula. Conditioning on a particular adaptation event answers a different question from averaging over all trial paths, so the two should not be confused.
Common Mistakes
- “Both stage estimates are unbiased, so their combination must be unbiased.” This is true for fixed weights, but not generally for weights that depend on the stage 1 estimate. The relevant quantity is \(\mathrm{E}\{W(\hat{\theta}_1-\theta)\}\).
- “The design changed, so bias must be present.” A change alone does not prove bias. A fixed weight or a weight independent of the interim estimation error gives zero bias under the stated assumptions.
- “Positive bias means the observed estimate is too high.” Bias describes the estimator’s average over repeated trials at a specified true effect. A particular estimate may be above or below that average.
- “An unbiased stage 2 estimate settles the issue.” It removes the stage 2 contribution in the derivation, but it does not remove the stage 1 weight-error association.
Check Your Understanding
Answer each question using the two-stage weighted-estimator model. Solutions follow the questions.
- If \(W=0.4\) is fixed and both stage estimates are unbiased, what is the bias of \(\hat{\theta}=W\hat{\theta}_1+(1-W)\hat{\theta}_2\)?
- Suppose \(\hat{\theta}_1\sim N(\theta,2^2)\), the threshold is \(c=\theta\), and the rule uses \(w_H=0.7\), \(w_L=0.3\). What is the bias?
- For the threshold rule, what happens to the bias sign if the high-interim group is assigned a smaller stage 1 weight than the low-interim group?
- Why is conditional unbiasedness of the stage 2 estimate needed for the bias identity?
Solutions. (1) The bias is \(0.4\mathrm{E}(\hat{\theta}_1-\theta)+0.6\mathrm{E}(\hat{\theta}_2-\theta)=0\). (2) Here \(a=(c-\theta)/s_1=0\), so the bias is \((0.7-0.3)(2)\phi(0)=(0.4)(2)(0.3989)=0.3192\) outcome units. (3) The difference \(w_H-w_L\) is negative, while \(s_1\phi(a)>0\), so the bias is negative. (4) It permits replacing the conditional expectation of \(\hat{\theta}_2\) by \(\theta\); otherwise the stage 2 term may contribute additional bias.