Observed Treatment Effects And Sample Size Bias

Why an Interim Effect Can Mislead

An interim estimate of treatment effect is based on only part of the planned trial data. It therefore has sampling variability: repeated trials with the same true effect can produce quite different interim estimates. This matters particularly when a sample-size rule uses the observed effect directly. For many standard designs, the required sample size is proportional to the inverse square of the effect used for planning. A noisy estimate in the denominator can consequently produce a very large change in the recommended sample size.

There is a second issue when the estimate influences whether or how the trial adapts. If a trial is selected for an update because its interim result looks promising, the selected estimate tends to be higher than the true effect, even when the estimate was unbiased before selection. These are distinct concerns: ordinary sampling variability makes recommendations unstable, while selection makes the estimates used in some decisions systematically optimistic.

A Planning Model and Its Assumptions

Consider a two-arm trial with a continuous outcome, equal allocation, independent observations, and a common known standard deviation \(\sigma\). Define \(\delta\) as the treatment-minus-control mean difference, with positive values representing benefit. At an interim analysis, suppose \(n_1\) participants per arm have complete outcome data. The difference in interim sample means, \(\hat{\delta}_1\), has the sampling distribution

This model requires independent outcomes with normally distributed errors and the stated common variance; the normal model for the difference in means is also a useful approximation in sufficiently large samples. For the calculations below, assume the standard deviation is known. If it is estimated, uncertainty in that estimate adds another source of variability, rather than removing the problem discussed here.

For a conventional two-sided test at significance level \(\alpha\), a commonly used normal-approximation planning formula for the required final number per arm is

Here \(1-\beta\) is the target power and \(\delta^*>0\) is the effect assumed for planning. The formula uses the usual normal approximation to achieve the target power; the small probability in the opposite tail is neglected in this approximation. The resulting \(n\) is a final sample size per arm, not the number of additional participants. A required integer sample size is rounded up.

Why Selection Creates Upward Bias

Before any adaptation decision, the normal model gives \(E(\hat{\delta}_1)=\delta\): the interim difference in means is unbiased. But suppose the trial is considered promising only when \(\hat{\delta}_1\) exceeds a threshold \(c\). Among trials passing that rule, interim estimates are not an unselected sample from their original distribution. Estimates below \(c\) have been excluded.

The conditional mean can be derived directly. Write \(\tau=\sqrt{2\sigma^2/n_1}\) and \(Z=(\hat{\delta}_1-\delta)/\tau\), so \(Z\sim N(0,1)\). Let \(a=(c-\delta)/\tau\). Then passing the rule is equivalent to \(Z>a\), and

For the last equality, the standard normal density \(\phi(z)\) satisfies \(\phi'(z)=-z\phi(z)\), so \(\int_a^\infty z\phi(z)\,dz=\phi(a)\); also \(P(Z>a)=1-\Phi(a)\). Dividing the integral by this probability gives the conditional mean. Since \(\phi(a)>0\) and \(1-\Phi(a)>0\), the conditional mean exceeds \(\delta\). This is a selection effect, not a change in the estimator’s unconditional expectation.

Instability From the Inverse-Square Rule

Even without selection, the mapping \(K/\hat{\delta}_1^2\) is highly nonlinear. When the estimate is positive and near zero, a small change in it can produce a very large change in the computed sample size. For example, halving the planning effect multiplies the calculated sample size by four. When the estimate is zero, the formula is undefined; when it is negative, using its signed value does not yield a meaningful positive-benefit sample-size calculation. Replacing it mechanically with its absolute value is not a general solution, because a large estimate in the unfavorable direction should not be mistaken for evidence of a large beneficial effect.

The nonlinearity also means that calculating a sample size from an average estimate is not the same as averaging sample sizes across possible estimates. In an untruncated normal model with positive density arbitrarily close to zero, the expected value of \(1/\hat{\delta}_1^2\) is infinite: near zero, the integral includes a positive density times \(1/x^2\), whose integral diverges. Real trial rules must define what happens for small or nonpositive estimates, but a rule that limits the recommendation changes the calculation and should be evaluated as part of the prespecified design.

How to Assess an Observed-Effect Rule

A responsible assessment separates the statistical estimate from the decision rule that uses it. Useful checks include:

• Compare the estimate with its standard error. An estimate only modestly larger than its standard error is especially vulnerable to substantial relative fluctuation.
• Inspect the rule over a range of estimates. Calculate recommendations for plausible values, including estimates near zero and values on the unfavorable side. Large jumps reveal sensitivity that a single example can hide.
• Account for selection. If only trials in a promising region trigger an increase, describe estimates conditional on entering that region as selected estimates, not as unbiased forecasts of the true effect.
• Check the rule before the trial begins. The analysis should specify how an estimate enters the update and what happens when it is near zero or has the unfavorable sign. A post hoc response to the observed result does not have the same interpretation as a prespecified rule.

A diagnostic is not a proof that a particular adaptation is valid. In particular, a promising observed effect does not establish that the trial is adequately powered after adaptation, and a recommendation computed from the plug-in formula does not by itself settle the properties of the final inference. The operating characteristics of the complete design must be assessed under the planned decision rule and relevant true effects.

Alternatives and Limitations

A common alternative is to base the sample-size calculation on a prespecified clinically relevant effect \(\delta^*\), rather than replacing it with the interim estimate. This avoids making the planning target move directly with a noisy estimate, but it does not ensure that the target is correct or that the trial will have the desired power if the true effect differs from it. The choice of \(\delta^*\) remains a substantive design assumption.

A blinded sample-size re-estimation based on a nuisance parameter, such as outcome variability, addresses a different problem: it updates a quantity that affects sample size without directly using the treatment contrast. It can avoid this particular source of treatment-effect selection bias, although it has its own assumptions and operating-characteristic questions. A conditional-power rule instead evaluates future success under specified effect assumptions; it should not be confused with treating one noisy observed effect as known truth. The earlier material on conditional power and promising-zone adaptation develops those decision frameworks.

Shrinkage or model-based effect estimates can reduce the influence of extreme interim results, but they require additional assumptions and do not automatically remove selection effects. Any alternative should be compared using the full trial design, including how interim information affects sample size and final analysis. The central limitation is that an interim effect estimate contains limited information about the true effect, while a direct inverse-square update can give that limited information disproportionate influence.

Common Mistake