Diagnosing Unstable Internal-Pilot Variances

Why a Point Estimate Can Be Misleading

An internal-pilot variance estimate may produce a single updated enrollment target, but that number can conceal substantial uncertainty. If modest changes in the estimated standard deviation lead to large changes in required enrollment, reporting only the point estimate gives a false impression of precision. The practical question is not just “What sample size does this estimate imply?” It is also “How much could that answer change because the interim variance estimate is uncertain?”

This tutorial develops a diagnostic for a continuous outcome in a two-arm trial. The focus is on uncertainty in the blinded variance estimate and the consequences of that uncertainty for enrollment—not on selecting an assumed treatment effect. The fixed-effect sample-size calculation is treated as a planning formula with its inputs held fixed except for the variance.

Working model and assumptions: Outcomes are independent and normally distributed within each treatment group, with a common variance \(\sigma^2\). The groups are allocated equally, the target difference \(\Delta\) is fixed, and the planned test is two-sided with significance level \(\alpha\) and power \(1-\beta\). The internal-pilot variance estimator must have a justified uncertainty calibration. A chi-square interval is exact for the ordinary pooled within-group variance under this model; it is not automatically exact for a blinded, treatment-effect-corrected estimator.

For equal allocation, a common normal-approximation planning formula for total enrollment is

N(\sigma^2) = \frac{4\sigma^2\left(z_{1-\alpha/2}+z_{1-\beta}\right)^2}{\Delta^2},

where \(z_q\) is the \(q\)-quantile of the standard normal distribution. Since \(N(\sigma^2)\) increases directly with \(\sigma^2\), uncertainty about the variance transfers directly into uncertainty about the required enrollment. In an equal-allocation design, the final total is rounded up to an even integer so that both groups have the same size.

From Variance Uncertainty to Enrollment Uncertainty

When a variance estimate \(V\) has the scaled-chi-square sampling distribution \(\nu V/\sigma^2\sim\chi^2_\nu\), a two-sided \(100(1-\gamma)\%\) confidence interval for \(\sigma^2\) is

\left[ \frac{\nu V}{\chi^2_{\nu,\,1-\gamma/2}}, \frac{\nu V}{\chi^2_{\nu,\,\gamma/2}} \right],

where \(\chi^2_{\nu,q}\) is the \(q\)-quantile of a chi-square distribution with \(\nu\) degrees of freedom. For an ordinary pooled within-group variance in a two-arm trial, \(\nu=n_1+n_2-2\). For example, 42 participants allocated 21 per group give \(21+21-2=40\) degrees of freedom—not 41 or 42.

A blinded variance estimator generally is not the ordinary pooled within-group variance, because the treatment labels are masked. If the estimator corrects a pooled blinded variance using an assumed treatment effect, its sampling distribution can differ from the simple scaled-chi-square distribution. In that case, use an interval based on a justified calibration for the actual estimator. A scaled-chi-square approximation can still be useful as a working diagnostic, but its coverage should not be called exact unless its assumptions are met. The earlier material on precision of variance estimates at interim looks provides the basis for understanding how degrees of freedom affect this uncertainty.

Diagnostic principle: Transform both endpoints of a valid or explicitly labeled working interval for \(\sigma^2\) through the sample-size formula. Because the formula is increasing in \(\sigma^2\), the lower variance endpoint gives the lower enrollment requirement and the upper endpoint gives the upper requirement. This range is a sensitivity summary; it is not a probability distribution for the final sample size.

A useful report therefore includes at least three quantities: the enrollment based on the interim point estimate, the enrollment range induced by variance uncertainty, and whether the range crosses an operational threshold such as a recruitment cap. If decisions are sensitive to the interval endpoint, the point estimate alone is not a reliable basis for a firm update.

Worked Example: A 42-Participant Internal Pilot

Hypothetical data and calibration. Suppose an internal pilot has enrolled 42 participants, 21 per group. The operational blinded variance estimate, after the trial’s prespecified correction, is \(V=144\ \text{(outcome units)}^2\). For this illustration, suppose its validated working uncertainty calibration is scaled chi-square with \(\nu=40\), matching the degrees-of-freedom benchmark for a pooled within-group variance with 21 participants per group. This assumption describes the calibration used for the example; it does not claim that a masked calculation is itself a within-group variance.

Plan for a two-sided \(\alpha=0.05\), 80% power, and a fixed target difference of \(\Delta=5\) outcome units. Using \(z_{0.975}=1.960\) and \(z_{0.80}=0.8416\), the point-estimate calculation is

N(144) = \frac{4(144)(1.960+0.8416)^2}{5^2} = 180.8.

The required equal-allocation total based on the point estimate is therefore 182 participants.

For a 95% variance interval, use \(\chi^2_{40,0.975}=59.342\) and \(\chi^2_{40,0.025}=24.433\):

\left[ \frac{40(144)}{59.342}, \frac{40(144)}{24.433} \right] = [97.06,\ 235.7]\ \text{(outcome units)}^2.

The corresponding standard-deviation interval is approximately \([9.852,\ 15.35]\) outcome units. Transforming the variance endpoints into total enrollment gives

N(97.06)=\frac{4(97.06)(2.8016)^2}{25}=121.9, \qquad N(235.7467)=\frac{4(235.7467)(2.8016)^2}{25}=296.1.

After rounding to even totals, the working enrollment range is 122 to 298 participants. Thus, the point estimate suggests 182 participants, but the variance uncertainty is wide enough to imply a substantially larger requirement. Under the stated calibration, this is a warning against presenting 182 as a stable, precise update.

Decision Sensitivity and Interim Sample Size

An interval is most informative when connected to the decision that must be made. For example, if a trial cannot recruit more than a specified number of participants, ask whether the sample-size requirement remains below that cap over the plausible variance range. Equivalently, calculate the largest variance compatible with the cap. This makes the sensitivity check interpretable in the original variance units.

Worked Example: Does an Enrollment Cap Make the Update Unreliable?

Continue with the hypothetical planning assumptions above and suppose the maximum feasible total enrollment is 200. The point estimate \(V=144\) implies 180.8 participants before equal-allocation rounding, or 182 after rounding, so the point estimate appears to fit the cap.

To find the variance at which the unrounded requirement reaches 200, rearrange the planning formula:

\sigma^2_{\mathrm{cap}} = \frac{N_{\mathrm{cap}}\Delta^2} {4(z_{1-\alpha/2}+z_{1-\beta})^2} = \frac{200(25)}{4(2.8016)^2} = 159.3\ \text{(outcome units)}^2.

The 95% working variance interval from the first example extends to \(235.7\), which is above \(159.3\). At that upper endpoint the planned requirement is about 296 participants, well beyond the cap. Therefore the feasibility conclusion depends on which plausible variance value is used: the point estimate fits the cap, but the uncertainty range does not. Under this diagnostic, the update is not robust enough to support an unqualified claim that the trial can achieve its target within 200 participants.

This does not prove that the true variance exceeds the cap threshold. A confidence interval does not assign a probability that the fixed parameter lies above or below a threshold. It shows that the data and calibration leave a range of variance values for which the operational conclusion changes.

Precision generally improves as the internal-pilot sample grows, because a variance estimate based on more information tends to have a tighter sampling distribution. The comparison must use the actual interim estimator and its calibration: the nominal number of enrolled participants is not always the estimator’s effective degrees of freedom.

Worked Example: Comparing Two Interim Sample Sizes

Consider a second hypothetical interim analysis with 102 participants, 51 per group. Suppose the blinded variance estimate is again \(144\ \text{(outcome units)}^2\), and the working scaled-chi-square calibration uses \(\nu=100\), the pooled within-group benchmark \(51+51-2\). The point-estimate enrollment remains 180.8 before rounding, or 182 participants.

Using \(\chi^2_{100,0.975}=129.561\) and \(\chi^2_{100,0.025}=74.222\), the 95% working interval is

\left[ \frac{100(144)}{129.561}, \frac{100(144)}{74.222} \right] = [111.1,\ 194.0]\ \text{(outcome units)}^2.

Transforming these endpoints gives approximate total enrollment requirements of

N(111.1445)\approx139.6, \qquad N(194.0)=243.6.

With equal-allocation rounding, the range is about 140 to 244 participants. It is narrower than the 122 to 298 range from the 42-participant interim analysis, while the point estimate is unchanged. Yet the upper endpoint still exceeds the 200-participant cap from the previous example. More interim information has improved precision, but it has not made the feasibility conclusion robust under this calibration.

Justification and Practical Checks

The transformation used above follows directly from monotonicity. Write \(C=4(z_{1-\alpha/2}+z_{1-\beta})^2/\Delta^2\), which is positive when \(\Delta\ne 0\). Then \(N(\sigma^2)=C\sigma^2\). If an interval for \(\sigma^2\) is \([L,U]\), multiplying its endpoints by the positive constant \(C\) preserves their order:

L\leq \sigma^2\leq U \quad\Longrightarrow\quad CL\leq N(\sigma^2)\leq CU.

This argument justifies the enrollment sensitivity range without requiring a separate probability model for the rounded sample size. Rounding should occur after calculating the endpoints. For equal allocation, round each required total up to the next even integer; rounding to the nearest integer could produce inadequate planned power or unequal group sizes.

Before relying on the diagnostic, check that the variance estimate is on the correct scale, that the degrees of freedom or calibration correspond to the actual estimator, and that the assumed effect, test, power, and allocation used in the sample-size formula are the intended planning inputs. Look for a wide interval relative to the point estimate, a large spread in the implied enrollment range, or an interval that crosses a feasibility threshold. These are signals that the proposed update is sensitive to variance uncertainty.

A point-estimate-only update is simple but hides uncertainty. A scenario grid—for example, calculating enrollment at several plausible standard deviations—is transparent and useful for communication, but its scenarios are not automatically confidence limits. An uncertainty interval gives a principled range when its sampling calibration is justified. If the calibration is only approximate, describe it as a working range and do not overstate its coverage.

Common Pitfalls and Limitations

Using the wrong degrees of freedom. For 21 participants in each group, the pooled within-group variance has \(21+21-2=40\) degrees of freedom. The total participant count is not the degrees of freedom for that estimator.
Treating a blinded correction as an ordinary pooled variance. A masked, treatment-effect-corrected estimate need not have the exact chi-square distribution of the within-group estimator. Use a calibration justified for the operational estimator.
Calling the transformed range a probability statement. A 95% interval with valid coverage is a confidence interval for the variance; the corresponding enrollment range is a transformed sensitivity range, not a 95% probability statement about the future sample size.
Inspecting only whether the point estimate meets a cap. If plausible variance values imply enrollment above the cap, the feasibility conclusion is sensitive even when the point estimate fits.
Ignoring assumptions in the planning formula. These calculations hold the target difference, allocation, significance level, and power fixed. They do not guarantee the design’s performance if those inputs or the variance model are inappropriate.

The diagnostic also cannot repair a poorly calibrated estimator or establish that the common-variance normal model is suitable. Its purpose is narrower: make the consequences of variance uncertainty visible before treating an interim point estimate as a dependable enrollment target.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A balanced internal pilot has 30 participants per group. What are the degrees of freedom for the ordinary pooled within-group variance?
If a valid variance interval is \([80,150]\) and the fixed sample-size formula is \(N(\sigma^2)=1.2\sigma^2\), what is the corresponding unrounded enrollment range?
A point estimate implies 160 participants, but the transformed variance interval implies requirements from 130 to 230. The recruitment cap is 200. Is the feasibility conclusion robust?
Why is a chi-square interval based on 40 degrees of freedom not automatically an exact interval for every blinded corrected variance estimate from a 42-participant pilot?

Solutions. (1) \(30+30-2=58\) degrees of freedom. (2) Since the multiplier is positive, the range is \([1.2(80),1.2(150)]=[96,180]\) participants before any design-specific rounding. (3) No. The upper implied requirement, 230, exceeds the cap, so the conclusion depends on the variance value within its uncertainty range. (4) The ordinary pooled within-group variance has a specific chi-square sampling distribution under the normal, common-variance model, but a blinded corrected estimator is a different statistic and may have a different distribution. Its uncertainty interval requires an appropriate calibration.