Pitfall: Treating A Blinded Pooled Estimate As Treatment-Neutral

Why Blinding Does Not Make a Pooled Estimate Treatment-Neutral

Blinding can prevent an interim team from seeing which treatment produced which outcomes. It does not remove the consequences of treatment assignment from the outcomes themselves. If the treatments produce different means or event rates, combining observations across arms can make the pooled distribution look more variable, or give it a different event rate, than either arm has on its own.

This matters when a blinded sample-size re-estimation treats a pooled estimate as though it were a neutral estimate of an arm-specific nuisance parameter. The pooled estimate may be useful, but its meaning depends on the quantity the update needs. An estimate of overall event accrual, for example, is not interchangeable with an estimate of the control-arm event rate.

Definition: A treatment-contaminated pooled estimate is a blinded estimate calculated across treatment groups that incorporates not only within-arm outcome variation, but also differences between the groups. Blinding hides group labels; it does not guarantee that the pooled estimate is unaffected by the treatment effect.

How Treatment Separation Enters a Pooled Variance

Suppose a continuous outcome has mean \(\mu_0\) and variance \(\sigma_0^2\) in the control arm, and mean \(\mu_1\) and variance \(\sigma_1^2\) in the treatment arm. Let \(q\) be the fraction assigned to treatment, and let \(\delta=\mu_1-\mu_0\). If we pool outcomes across arms, the variance of the resulting mixture is

\sigma_{\mathrm{pool}}^2 =(1-q)\sigma_0^2+q\sigma_1^2+q(1-q)\delta^2.

The first two terms are the weighted average of the within-arm variances. The last term is the between-arm contribution. It is nonnegative and is zero only when the arm means are equal or one arm has no allocation. With equal allocation and equal within-arm variances \(\sigma^2\), this reduces to \(\sigma_{\mathrm{pool}}^2=\sigma^2+\delta^2/4\).

This is a variance of the combined outcome distribution, not the residual variance used to compare the arms after accounting for treatment. A pooled sample variance can therefore be a valid estimate of overall marginal variability while still being the wrong estimate for a rule that needs within-arm variability.

Justification: Let \(T\) denote treatment group and \(Y\) the outcome. The law of total variance gives \(\operatorname{Var}(Y)=E[\operatorname{Var}(Y\mid T)]+\operatorname{Var}(E[Y\mid T])\). The first term is \((1-q)\sigma_0^2+q\sigma_1^2\). The second is the variance of a two-point variable taking values \(\mu_0,\mu_1\) with probabilities \(1-q,q\), which equals \(q(1-q)(\mu_1-\mu_0)^2\). Adding the terms gives the pooled-variance formula.

For fixed arm sizes \(n_0\) and \(n_1\), the same point appears in the expected ordinary sample variance about the grand mean:

E[s_{\mathrm{pool}}^2] = \frac{(n_0-1)\sigma_0^2+(n_1-1)\sigma_1^2+ \frac{n_0n_1}{n_0+n_1}(\mu_1-\mu_0)^2} {n_0+n_1-1}.

The expression includes a term for treatment separation even though the analyst calculating \(s_{\mathrm{pool}}\) need not know which observations came from which arm. Thus, “blinded” describes access to assignments, not a guarantee of treatment-neutrality.

Worked Example: A Pooled Continuous Variance Inflates the Update

Suppose an interim sample has 50 participants per arm. The hypothetical outcomes have a common within-arm standard deviation of 10 points and an arm-mean difference of 8 points. Consider a normal-approximation planning calculation for a two-sided test with \(\alpha=0.05\), 90% power, equal allocation, and target difference \(\delta_{\mathrm{target}}=5\) points. Assume independent outcomes, a common within-arm variance, and that the target difference remains the design effect.

With \(q=0.5\), the expected pooled variance is approximately

\sigma_{\mathrm{pool}}^2 =10^2+(0.5)(0.5)(8^2) =100+16 =116\ \text{points}^2.

For this illustrative planning approximation, total sample size is \(N=4(z_{1-\alpha/2}+z_{0.90})^2\sigma^2/\delta_{\mathrm{target}}^2\), where \(z_{0.975}=1.960\) and \(z_{0.90}=1.282\). Using the pooled estimate gives

N =\frac{4(1.960+1.282)^2(116)}{5^2} \approx 195.0.

Using the within-arm variance instead gives \(N=4(1.960+1.282)^2(100)/25\approx168.1\). Rounded up, these calculations imply 196 and 169 participants, respectively. Under the stated assumptions, the pooled-variance update is conservative with respect to this variance component: it calls for a larger trial than the calculation using within-arm variance. It may also be inefficient if the extra variance is due to the treatment contrast rather than greater within-arm noise. This approximation illustrates the mechanism; an operational design must use the test-specific power calculation and its prespecified assumptions.

Binary Outcomes: The Pooled Rate Is a Mixture

For a binary outcome, let \(p_0\) and \(p_1\) be the event probabilities in the control and treatment arms. The pooled event probability under allocation fraction \(q\) is \(p_{\mathrm{pool}}=(1-q)p_0+qp_1\). Its Bernoulli variance obeys the same decomposition:

p_{\mathrm{pool}}(1-p_{\mathrm{pool}}) =(1-q)p_0(1-p_0)+qp_1(1-p_1) +q(1-q)(p_1-p_0)^2.

The pooled Bernoulli variance includes a between-arm term. In particular, it is at least as large as the weighted average of the two within-arm Bernoulli variances. But the pooled event probability itself is not automatically too high or too low for every use: it is a weighted average of the arm-specific rates. Whether it is suitable depends on whether the update needs an overall rate or an arm-specific rate.

Worked Example: Pooled Binary Variance Versus Arm-Specific Variance

Consider a hypothetical equally allocated trial with event probabilities \(p_0=0.20\) and \(p_1=0.40\). Assume independent binary outcomes and use a normal-approximation planning proxy that sets the effect divided by its standard error equal to \(z_{0.975}+z_{0.90}=3.242\). The target risk difference is 0.20.

The pooled rate is \(p_{\mathrm{pool}}=(0.20+0.40)/2=0.30\), so its Bernoulli variance is \(0.30(0.70)=0.21\). The weighted within-arm variance is \([0.20(0.80)+0.40(0.60)]/2=(0.16+0.24)/2=0.20\). The remaining \(0.01\) is the between-arm term: \((0.5)(0.5)(0.20)^2=0.01\).

If \(N\) is the total sample size, the variance of the difference in sample proportions using the arm-specific variances is \(2(0.16+0.24)/N=0.80/N\). The proxy therefore gives

N=\frac{(3.242)^2(0.80)}{(0.20)^2} \approx210.2,

or 211 participants after rounding up. If the pooled variance \(0.21\) is substituted as though it were the common variance in both arms, the variance becomes \(4(0.21)/N=0.84/N\), giving \(N=(3.242)^2(0.84)/(0.20)^2\approx220.7\), or 221 participants. Here the substitution is conservative for the displayed proxy. Exact sample sizes depend on the chosen test and its power calculation; this example isolates the variance contribution rather than claiming one formula is universal.

Event Rates: Match the Estimate to the Target

In a fixed follow-up window, the overall probability of an event under the trial’s allocation is \(p_{\mathrm{pool}}=(1-q)p_0+qp_1\). If the update forecasts the total number of events among participants randomized under that same allocation and follow-up, this marginal rate can be exactly the relevant quantity. It is still not treatment-neutral: a beneficial treatment can lower it, and a harmful treatment can raise it.

If the update instead uses the pooled rate as a proxy for the control event rate, the direction of error depends on the treatment effect. When treatment reduces events, \(p_{\mathrm{pool}}<p_0\), so using the pooled rate as the control rate can overstate the enrollment needed for a control-specific target. When treatment increases events, \(p_{\mathrm{pool}}>p_0\), the same substitution can understate that enrollment. Similar care is needed when translating event probabilities into follow-up or accrual projections: the rate’s meaning depends on the population, allocation, and time window to which it refers.

Worked Example: Overall Survival Events Under Equal Allocation

Suppose a hypothetical survival trial allocates participants equally and uses a fixed follow-up window. By that window, the event probabilities are estimated to be 0.60 in control and 0.30 in treatment. Assume the rates apply to the same population and follow-up definition used in the projection, and that the target is 180 total events among randomized participants.

The relevant overall event probability is \(p_{\mathrm{pool}}=(0.60+0.30)/2=0.45\). The projected enrollment is \(180/0.45=400\) participants. If someone incorrectly used the control rate of 0.60 as the overall rate, the projection would be \(180/0.60=300\) participants. At 300 participants under equal allocation, the expected total events are \(300(0.45)=135\), not 180.

In this example, the pooled rate is treatment-affected but appropriate for forecasting total events under the specified allocation. It is not an estimate of the control-arm rate. If instead the treatment event probability were 0.60 and the control probability 0.30, the overall rate would be 0.45 again; using the control rate 0.30 would project 600 participants, an inefficient overestimate for the total-event target. The relevant rate must therefore be chosen by the projection’s target, not by whether it is blinded.

When the Distortion Is Conservative, Inefficient, or Risky

A variance-based update is conservative in the narrow sample-size sense when it uses a variance larger than the variance relevant to its power calculation and consequently increases \(N\). The continuous and binary examples show one common mechanism: separation between treatment means or rates adds a nonnegative between-arm component to the pooled variance. This does not mean every pooled-variance update is automatically conservative overall. The required variance may differ by arm, the planned analysis may use adjustment, or the actual data may depart from the assumed model.

A pooled event rate can instead cause either direction of error when it is used for an arm-specific projection. For total events under the same allocation and follow-up window, the mixture rate may be the right projection input. For control-only events, or for a projection under a different allocation, it generally is not. A rate measured at one follow-up duration also cannot be treated as the rate at another duration without a suitable model.

Practical checks include the following:

State the target quantity. Is the update estimating within-arm outcome variance, overall event accrual, or an arm-specific event rate?
Write down the allocation mixture. For outcomes combined across arms, identify the assumed or realized value of \(q\), and check whether it matches the allocation used for the projection.
Stress-test plausible treatment effects. Recalculate the pooled variance or rate over clinically plausible arm differences. A large change indicates that the nominally blinded estimate is effect-sensitive.
Check the outcome definition and time window. Different follow-up periods, censoring patterns, or endpoint definitions can make apparently similar event rates incomparable.
Prespecify how the estimate is used. A blinded rule should state which pooled quantity drives the update, the target it represents, and any bounds or alternative assumptions used if it is unstable.

When preserving the blind is important, the design can use sensitivity scenarios, external nuisance information, or an estimate that directly targets the overall quantity needed for the update. Any rule should be evaluated under plausible treatment effects, allocation ratios, and nuisance parameters. Blinding can still be valuable; the key is not to mistake concealment of assignments for absence of treatment-effect influence.

Common Mistake

Wrong reasoning: “The interim statistician cannot see treatment labels, so the pooled standard deviation or event rate must estimate a treatment-neutral nuisance parameter.”

Correction: A pooled variance includes between-arm separation, and a pooled event rate is an allocation-weighted average of arm-specific rates. Determine which quantity the re-estimation actually needs, then assess whether the pooled estimate targets it under the trial’s allocation and follow-up assumptions. A larger pooled variance may inflate the planned sample size; a pooled rate may overstate or understate an arm-specific rate.

Key takeaway: Blinded does not mean treatment-neutral. For variance updates, distinguish within-arm variability from between-arm separation. For event-rate updates, distinguish an overall allocation-weighted rate from an arm-specific rate.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

With equal allocation, common within-arm variance \(\sigma^2\), and arm-mean difference \(\delta\), what is the pooled variance? What happens to it when \(\delta=0\)?
In an equally allocated binary trial, \(p_0=0.10\) and \(p_1=0.30\). Calculate the pooled rate, the weighted average within-arm variance, and the between-arm contribution.
Under equal allocation, a fixed-window event probability is 0.50 in control and 0.20 in treatment. What rate should be used to project total events under the same allocation? Is it a control-arm rate?
Explain why an increase in pooled variance can produce an inefficient sample-size update without implying that the treatment effect has increased within-arm noise.

Solutions. (1) The pooled variance is \(\sigma^2+\delta^2/4\); when \(\delta=0\), it equals the within-arm variance \(\sigma^2\). (2) The pooled rate is \((0.10+0.30)/2=0.20\). The weighted within-arm variance is \([0.10(0.90)+0.30(0.70)]/2=(0.09+0.21)/2=0.15\). The between-arm contribution is \((0.5)(0.5)(0.20)^2=0.01\), giving pooled variance \(0.16\). (3) Use the overall rate \((0.50+0.20)/2=0.35\) for total events under the same allocation and follow-up. It is not the control rate, which is 0.50. (4) The pooled variance can rise because the two arm means differ, through the between-arm term, even if each arm’s own variance is unchanged. If the sample-size calculation needs within-arm variance, treating this larger pooled value as within-arm noise can call for more participants than necessary.