Confidence Intervals With Estimated Variance

Why Estimating Variances Matters After Adaptation

An interval based on a known variance can use a standard normal quantile when its estimator is normal. In practice, the variances are usually estimated. That changes the distribution of the interval statistic, and adaptation can change it again: if the stage 1 variance estimate determines the stage 2 sample size, the stage 2 variance estimate has a different sampling distribution on different branches of the design.

The issue is not resolved merely by substituting estimated variances into a standard error. A plug-in interval may be useful as an approximation, but its coverage needs justification under the actual adaptation rule. This tutorial develops a conservative variance-adjusted interval for a two-stage normal model and makes clear which coverage guarantee it provides.

A Two-Stage Normal Model

Suppose the target is a common treatment effect \(\theta\), estimated in two independent stages. For stage \(j\), observations are normally distributed with mean \(\theta\) and variance \(\sigma_j^2\). The variances may differ between stages, and both are unknown. Write \(\bar X_j\) for the stage mean and \(s_j^2\) for its sample variance. Within each stage, observations are independent and identically distributed, and the stage mean is independent of the sample variance.

Stage 1 has fixed size \(n_1\). An adaptation rule uses \(s_1^2\) to select the stage 2 size \(n_{2,b}\), where \(b\) indexes the branch. Conditional on the selected size, stage 2 observations are generated independently of stage 1. The stage 2 sample variance therefore has a chi-square distribution conditional on the selected branch, with degrees of freedom determined by that branch:

If \(n_{2,b}\) varies across branches, these conditional distributions differ. Consequently, stage 2 data being generated independently of stage 1 does not imply that \(s_2^2\) is unconditionally independent of stage 1: its distribution depends on the stage 1-based branch. The conditional statement is the appropriate one.

For clarity, consider a prespecified combined estimator \(\hat\theta=a_1\bar X_1+a_2\bar X_2\), where \(a_1+a_2=1\) and the coefficients are fixed, not chosen from the observed stage 1 mean. Conditional on branch \(b\), its variance is \(V_b=a_1^2\sigma_1^2/n_1+a_2^2\sigma_2^2/n_{2,b}\). Because stage 1 normal sample mean and sample variance are independent, selection based on \(s_1^2\) does not truncate \(\bar X_1\). Under the model, \(\hat\theta-\theta\), conditional on branch, is normal with mean zero and variance \(V_b\). This setup isolates how variance estimation and branch-dependent sample size affect interval coverage; it does not cover every possible adaptive estimator.

Why the Usual Plug-In Interval Is Not Exact

A common plug-in standard error is \(\widehat{SE}_b=\sqrt{a_1^2s_1^2/n_1+a_2^2s_2^2/n_{2,b}}\). One may place a normal critical value around \(\hat\theta\). But the estimated variance in this expression is random, and its distribution is not generally a single scaled chi-square variable. In addition, conditional on a branch selected using \(s_1^2\), the stage 1 variance estimate has a truncated distribution. Thus the ratio of estimation error to plug-in standard error is not generally standard normal or an ordinary \(t\) statistic.

A Conservative Variance-Adjusted Interval

One way to protect unconditional coverage is to form upper confidence bounds for each variance, using the correct stagewise degrees of freedom, and then use those bounds to construct a conservative standard error. For any chi-square degrees of freedom \(d\), let \(q_{\gamma,d}\) be the lower \(\gamma\) quantile, defined by \(P(\chi^2_d\le q_{\gamma,d})=\gamma\). An upper confidence bound for a variance based on sample variance \(s^2\) and \(d\) degrees of freedom is \(d s^2/q_{\gamma,d}\), since the event that this bound is at least the true variance is equivalent to \(\chi^2_d\ge q_{\gamma,d}\), with probability \(1-\gamma\).

For stage 1, \(d_1=n_1-1\). Its usual chi-square bound has unconditional coverage \(1-\gamma_1\). It need not have that coverage conditional on the selected branch, because the branch was chosen using \(s_1^2\). For stage 2, use the branch-specific degrees of freedom \(d_{2,b}=n_{2,b}-1\) and a lower quantile for those degrees of freedom. Conditional on branch \(b\), this stage 2 bound has coverage \(1-\gamma_2\). Thus its failure probability is at most \(\gamma_2\) unconditionally as well, by averaging over branches.

By the union bound, the probability that either variance bound fails is no greater than \(\gamma_1+\gamma_2\). When both bounds hold, the true \(V_b\) is no greater than \(SE_{U,b}^2\). Conditional on the branch and variance estimates, the standardized estimation error \((\hat\theta-\theta)/\sqrt{V_b}\) is standard normal and independent of the variance estimates under this model. Therefore a central normal event with probability \(1-\alpha_*\), together with both variance bounds holding, guarantees that \(\theta\) is in the interval. The resulting unconditional coverage is at least \((1-\alpha_*)(1-\gamma_1-\gamma_2)\).

For a target coverage of at least \(1-\alpha\), one convenient conservative allocation is \(\alpha_*+\gamma_1+\gamma_2\le\alpha\), since \((1-\alpha_*)(1-\gamma_1-\gamma_2)\ge1-\alpha_*-\gamma_1-\gamma_2\). The price of this protection can be a wide interval, particularly with small samples or very conservative variance-bound quantiles.

Branch Dependence, Diagnostics, and Alternatives

Consider a rule that chooses \(n_2=6\) on one branch and \(n_2=10\) on another. Then \(s_2^2\), conditional on the first branch, has a scaled \(\chi^2_5\) distribution; conditional on the second, it has a scaled \(\chi^2_9\) distribution. For example, if \(\sigma_2^2=4\), the conditional variances of \(s_2^2\) are \(2(4^2)/5=6.400\) and \(2(4^2)/9=3.556\), respectively. The stage 2 observations are generated independently of stage 1, but the distribution of their variance estimate changes with the branch. This is why using a single, unqualified stage 2 variance distribution can give the wrong uncertainty calculation.

The conservative interval above targets unconditional coverage. It does not promise the same coverage after conditioning on each branch. In particular, the stage 1 variance bound is not calibrated conditionally on a branch defined using that same variance estimate. A conditional guarantee requires accounting for the truncated distribution induced by the rule; its calibration generally depends on the candidate variance and may require numerical inversion. Do not describe an unconditional guarantee as conditional coverage.

Other approaches have different trade-offs. A plug-in normal interval is simple but generally approximate. A Satterthwaite approximation replaces a weighted sum of estimated variance components by a scaled chi-square distribution and uses an effective degrees-of-freedom calculation; this can be useful, but branch selection and truncation must be incorporated if conditional accuracy is sought. Directly calibrating a branch-aware statistic under the full model can target conditional or unconditional coverage more closely, at the cost of design-specific calculations. The conservative upper-bound method is transparent and has a clear coverage argument, but often sacrifices interval length.

Useful diagnostics include checking the prespecified adaptation rule, the actual branch-specific sample sizes, residual patterns within each stage, and whether the normal and constant-within-stage variance assumptions are plausible. A variance estimate near zero, severe outliers, or a rule that also selects on the stage 1 mean are warnings that the stated model or coverage argument may not apply. Small-sample exact procedures and simulation-based assessment can be useful in such cases, but they must represent the adaptation mechanism rather than treating the realized sample sizes as if they had been fixed in advance.